Offline data development task log intelligent monitoring system and method based on large model

By adopting an intelligent monitoring system for offline data development task logs based on large models in software development, intelligently monitoring and controlling dirty data processing, the problem of dirty data affecting the stability of synchronization tasks is solved, the success rate and efficiency of data synchronization are improved, and the improvement of the success rate and efficiency of data synchronization is improved. Software development capabilities.

CN119025375AActive Publication Date: 2024-11-26JIANGSU ZHENYUN TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411069156.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-26
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

During the offline data synchronization process of software development, the generation of dirty data affects the stability and success rate of synchronization tasks. The existing technology relies on administrators to manually set dirty data thresholds, resulting in task failure or data abandonment, affecting synchronization capabilities.

Method used

Using an intelligent monitoring system for offline data development task logs based on large models, the software development business process name, configuration synchronization network links, select resource groups, build data associations and test connectivity, and use large models to build dirty data processing nodes to achieve the Intelligent monitoring and automated control of the amount of dirty data, start and stop synchronization tasks.

Benefits of technology

It reduces the risk of error reporting during offline data development task log synchronization, improves write success rate and efficiency, realizes digital offline synchronization level, and improves software development capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025375A_ABST
    Figure CN119025375A_ABST
Patent Text Reader

Abstract

The invention discloses an off-line data development task log intelligent monitoring system and method based on a large model, and belongs to the technical field of software development intelligent monitoring. The system comprises a data integration directory module, an offline synchronization module, a production environment operation and maintenance management module and an intelligent monitoring module. The output end of the data integration directory module is connected with the input end of the offline synchronization module; the output end of the offline synchronization module is connected with the input end of the production environment operation and maintenance management module; the output end of the production environment operation and maintenance management module is connected with the input end of the intelligent monitoring module. In the software development process, data synchronization of offline tasks can be achieved, the number nodes of dirty data are constructed according to a large model under sample data, automatic monitoring and management are formed, and the error reporting risk is reduced in the synchronous writing process of offline data development task logs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of software development intelligent monitoring, and in particular to an offline data development task log intelligent monitoring system and method based on a large model. Background Art

[0002] During the offline data synchronization process of software development, the data of the source data source must be able to be written to the destination data source, that is, the source data type needs to match the write data type. However, during the offline task process, a large amount of dirty data is usually generated, that is, an exception occurs during the writing of a single data to the target data source, and this data is dirty data. In general, data that fails to be written is classified as dirty data. In the actual operation process, the administrator can set the threshold of the number of dirty data generated through the task to ensure that data synchronization is completed when the synchronization task is configured in the data integration process. When the number of dirty data generated exceeds the threshold, the task will fail to exit. For example, if the number of dirty data allowed is set to 0, the task will fail to exit when dirty data is generated. When the number of dirty data generated is less than the threshold, the task will continue to run, but the dirty data will be discarded and not written to the destination.

[0003] The number of dirty data tasks is usually set manually by the administrator. However, during the learning process of a large model, due to the different properties of offline data, using a stable number of tasks will also affect the ability to synchronize data, resulting in various types of errors. Summary of the invention

[0004] The purpose of the present invention is to provide an offline data development task log intelligent monitoring system and method based on a large model to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for intelligent monitoring of offline data development task logs based on a large model, the method comprising the following steps:

[0006] S1. Obtain the name of the software development business process, drag and drop the offline synchronization node in the data integration directory module, and build an offline synchronization task;

[0007] S2. Under the offline synchronization task, configure the synchronization network link, select the data source and data destination of the offline synchronization task, and the resource group used to execute the synchronization task, test connectivity, edit the script, and configure the offline synchronization task;

[0008] S3. Import the offline synchronization task into the production environment operation and maintenance center, and the production environment operation and maintenance center calls the large model to configure the dirty data processing node;

[0009] S4. Based on the dirty data processing node and the offline synchronization task, the offline data development task log is intelligently monitored to control the start and stop of the synchronization task of the offline data development task log.

[0010] According to the above technical solution, in step S2, it also includes:

[0011] Select the switch network segment bound to the resource group, the EIP of the old resource group itself, or the EIP of the new universal resource group bound to the VPC configuration, and add them to the whitelist of data sources;

[0012] Build a data association between the data source and DataWorks and add a DataWorks data source to test connectivity. Specifically, the following steps are required:

[0013] In the Data Integration Catalog module, log in to the DataWorks console, select Data Integration in the navigation bar, select the corresponding workspace, enter Data Integration, select Data Source in the navigation bar, and then add a data source in the data source list. Set the newly added data source as the current data source and configure related connection parameters to test connectivity.

[0014] The configuration offline synchronization task also includes:

[0015] The periodically scheduled offline synchronization task needs to configure the related attributes when the task is automatically scheduled, and the related attributes include: configuring node scheduling attributes, configuring time attributes, and configuring resource attributes;

[0016] The configuration node scheduling attributes include: assigning scheduling parameters to custom variables, and supporting the assignment of constants;

[0017] The configuration time attributes include: a periodic scheduling method for defining tasks in a production environment;

[0018] The configuration resource attributes include: in a periodically scheduled offline synchronization task scenario, defining a scheduling resource group used when executing resources for a data integration task.

[0019] According to the above technical solution, in step S3, it also includes:

[0020] Based on different resource groups that perform synchronization tasks, large models under different resource groups are constructed respectively. The method for constructing a large model under any resource group includes:

[0021] Construct a sample data set, select the original data under the same resource group, the original data refers to the manual setting data of the user administrator, and form a specific data set ,in Refers to the offline synchronization task feature group, specifically including the total number of offline synchronization tasks and the number of associations between offline synchronization tasks. The association refers to the data interaction between any two offline synchronization tasks, which is defined as the number of associations at one time; Refers to configuring the number of dirty data processing nodes; using a fixed weight format Assign values, including:

[0022]

[0023] in, Refers to the weight set by the system; , Represents the normalized data of the total number of offline synchronization tasks and the number of associations between offline synchronization tasks;

[0024] A frequency histogram is constructed based on the sample data set. The area of ​​the rectangle in the frequency histogram is defined as the interval frequency, and the height of the rectangle is the average frequency density of the interval. Then:

[0025]

[0026] in, Refers to sample data i in the sample data set; Refers to the probability density function based on a sample data set; Refers to the cumulative distribution function based on the sample data set; h represents the bandwidth of the kernel density estimate, which is a system setting; , Respectively represent , The cumulative distribution function value under , that is, for continuous functions, all values ​​less than or equal to The value of , and its probability of occurrence is The cumulative distribution function value under ; all values ​​less than or equal to The value of , and its probability of occurrence is The cumulative distribution function value under ;

[0027] In a specific data set In the two-dimensional scenario, for any new data x, find its closest range in the sample data set ; Form the probability density function calculation for any new data x:

[0028]

[0029] Among them, s represents the sequence number; n represents the total number of samples in the sample data set; Represents the uniform kernel function; Represents x and distance;

[0030] According to the total number of offline synchronization tasks and the number of associations between offline synchronization tasks monitored by the system, a specific new data set is formed, thereby outputting the number of configured dirty data processing nodes under the specific new data set.

[0031] According to the above technical solution, it also includes:

[0032] Obtain offline synchronization tasks and the number of dirty data processing nodes configured under the offline synchronization tasks, and perform intelligent monitoring of offline data development task logs. If the number of dirty data generated within the period under the configured time attribute is lower than or equal to the number of dirty data processing nodes configured, the synchronization task of the offline data development task log is performed normally; if the number of dirty data generated within the period under the configured time attribute is higher than the number of dirty data processing nodes configured, the synchronization task of the offline data development task log is stopped, the synchronized offline task data is retained, and an error is reported on the administrator port at the same time.

[0033] An offline data development task log intelligent monitoring system based on a large model, the system includes: a data integration directory module, an offline synchronization module, a production environment operation and maintenance management module and an intelligent monitoring module;

[0034] The data integration directory module is used to obtain the name of the software development business process, drag the offline synchronization node in the data integration directory module, and build an offline synchronization task; the offline synchronization module is used to configure the synchronization network link under the offline synchronization task, select the data source and data destination of the offline synchronization task, and the resource group used to execute the synchronization task, test connectivity, edit the script, and configure the offline synchronization task; the production environment operation and maintenance management module is used to import the offline synchronization task into the production environment operation and maintenance center, and the production environment operation and maintenance center calls the large model to configure the dirty data processing node; the intelligent monitoring module is used to perform intelligent monitoring of the offline data development task log according to the dirty data processing node and the offline synchronization task, and control the start and stop of the synchronization task of the offline data development task log;

[0035] The output end of the data integration directory module is connected to the input end of the offline synchronization module; the output end of the offline synchronization module is connected to the input end of the production environment operation and maintenance management module; the output end of the production environment operation and maintenance management module is connected to the input end of the intelligent monitoring module.

[0036] According to the above technical solution, the data integration directory module includes a software development business unit and an offline synchronization task construction unit;

[0037] The software development business unit is used to record the software development business process name and form a software development business list; the offline synchronization task construction unit is used to drag the offline synchronization node and construct the offline synchronization task;

[0038] The output end of the software development business unit is connected to the input end of the offline synchronization task construction unit.

[0039] According to the above technical solution, the offline synchronization module includes an offline synchronization test unit and an offline synchronization configuration unit;

[0040] The offline synchronization test unit is used to configure the synchronization network link under the offline synchronization task, select the data source and data destination of the offline synchronization task, and the resource group used to execute the synchronization task, and test the connectivity; the offline synchronization configuration unit is used to edit the script, and configure the relevant attributes of the automatic scheduling of the task under the periodically scheduled offline synchronization task, and the relevant attributes include: configuration node scheduling attributes, configuration time attributes, and configuration resource attributes;

[0041] The output end of the offline synchronization test unit is connected to the input end of the offline synchronization configuration unit.

[0042] According to the above technical solution, the production environment operation and maintenance management module includes a production environment operation and maintenance center and a large model analysis unit;

[0043] The production environment operation and maintenance center is used to import offline synchronization tasks and call large model processing; the large model analysis unit forms large models under different resource groups based on sample data;

[0044] The output end of the large model analysis unit is connected to the input end of the production environment operation and maintenance center.

[0045] According to the above technical solution, the intelligent monitoring module includes an intelligent monitoring unit and an automatic start-stop control unit;

[0046] The intelligent monitoring unit is used to perform intelligent monitoring of the offline data development task log according to the dirty data processing node and the offline synchronization task, and control the start and stop of the synchronization task of the offline data development task log; the automatic start and stop unit is used to obtain the offline synchronization task and the number of configured dirty data processing nodes under the offline synchronization task, and perform intelligent monitoring of the offline data development task log. If the number of dirty data generated within the period under the configured time attribute is lower than or equal to the number of configured dirty data processing nodes, the synchronization task of the offline data development task log is performed normally; if the number of dirty data generated within the period under the configured time attribute is higher than the number of configured dirty data processing nodes, the synchronization task of the offline data development task log is stopped, the synchronized offline task data is retained, and an error is reported on the administrator port at the same time.

[0047] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention can realize data synchronization of offline tasks during the software development process, and build a number of nodes for dirty data based on the big model under the sample data to form automated monitoring and management, and reduce the risk of errors in the synchronous writing of offline data development task logs, improve the writing success rate and efficiency, achieve a digital offline synchronization level, and improve software development capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0049] Figure 1 It is a flow chart of an intelligent monitoring method for offline data development task log based on a large model of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] See also Figure 1 ,In the first embodiment, a method for intelligent monitoring of offline data development task log based on a large model is provided, the method includes: obtaining the name of the software development business process, dragging the offline synchronization node in the data integration directory module, and building the offline synchronization task; under the offline synchronization task, configuring the synchronization network link, selecting the data source and data destination of the offline synchronization task, and the resource group for executing the synchronization task, testing connectivity, editing the script, and configuring the offline synchronization task;

[0052] Select the switch network segment bound to the resource group, the EIP of the old resource group itself, or the EIP of the new universal resource group bound to the VPC configuration, and add them to the whitelist of data sources;

[0053] Build a data association between the data source and DataWorks and add a DataWorks data source to test connectivity. Specifically, the following steps are required:

[0054] In the Data Integration Catalog module, log in to the DataWorks console, select Data Integration in the navigation bar, select the corresponding workspace, enter Data Integration, select Data Source in the navigation bar, and then add a data source in the data source list. Set the newly added data source as the current data source and configure related connection parameters to test connectivity.

[0055] The configuration offline synchronization task also includes:

[0056] The periodically scheduled offline synchronization task needs to configure the related attributes when the task is automatically scheduled, and the related attributes include: configuring node scheduling attributes, configuring time attributes, and configuring resource attributes;

[0057] The configuration node scheduling attributes include: assigning scheduling parameters to custom variables, and supporting the assignment of constants;

[0058] The configuration time attributes include: a periodic scheduling method for defining tasks in a production environment;

[0059] The configuration resource attributes include: in a periodically scheduled offline synchronization task scenario, defining a scheduling resource group used when executing resources for a data integration task.

[0060] Import offline synchronization tasks into the production environment operation and maintenance center, and the production environment operation and maintenance center calls the large model to configure dirty data processing nodes;

[0061] Based on different resource groups that perform synchronization tasks, large models under different resource groups are constructed respectively. The method for constructing a large model under any resource group includes:

[0062] Construct a sample data set, select the original data under the same resource group, the original data refers to the manual setting data of the user administrator, and form a specific data set ,in Refers to the offline synchronization task feature group, specifically including the total number of offline synchronization tasks and the number of associations between offline synchronization tasks. The association refers to the data interaction between any two offline synchronization tasks, which is defined as the number of associations at one time; Refers to configuring the number of dirty data processing nodes; using a fixed weight format Assign values, including:

[0063]

[0064] in, Refers to the weight set by the system; , Represents the normalized data of the total number of offline synchronization tasks and the number of associations between offline synchronization tasks;

[0065] A frequency histogram is constructed based on the sample data set. The area of ​​the rectangle in the frequency histogram is defined as the interval frequency, and the height of the rectangle is the average frequency density of the interval. Then:

[0066]

[0067] in, Refers to sample data i in the sample data set; Refers to the probability density function based on a sample data set; Refers to the cumulative distribution function based on the sample data set; h represents the bandwidth of the kernel density estimate, which is a system setting; , Respectively represent , The cumulative distribution function value under , that is, for continuous functions, all values ​​less than or equal to The value of , and its probability of occurrence is The cumulative distribution function value under ; all values ​​less than or equal to The value of , and its probability of occurrence is The cumulative distribution function value under ;

[0068] In a specific data set In the two-dimensional scenario, for any new data x, find its closest range in the sample data set ; Form the probability density function calculation for any new data x:

[0069]

[0070] Among them, s represents the sequence number; n represents the total number of samples in the sample data set; Represents the uniform kernel function; Represents x and distance;

[0071] According to the total number of offline synchronization tasks and the number of associations between offline synchronization tasks monitored by the system, a specific new data set is formed, thereby outputting the number of configured dirty data processing nodes under the specific new data set.

[0072] Also includes:

[0073] Obtain offline synchronization tasks and the number of dirty data processing nodes configured under the offline synchronization tasks, and perform intelligent monitoring of offline data development task logs. If the number of dirty data generated within the period under the configured time attribute is lower than or equal to the number of dirty data processing nodes configured, the synchronization task of the offline data development task log is performed normally; if the number of dirty data generated within the period under the configured time attribute is higher than the number of dirty data processing nodes configured, the synchronization task of the offline data development task log is stopped, the synchronized offline task data is retained, and an error is reported on the administrator port at the same time.

[0074] In the second embodiment, a large model-based offline data development task log intelligent monitoring system is provided, which includes: a data integration directory module, an offline synchronization module, a production environment operation and maintenance management module, and an intelligent monitoring module;

[0075] The data integration directory module is used to obtain the name of the software development business process, drag the offline synchronization node in the data integration directory module, and build an offline synchronization task; the offline synchronization module is used to configure the synchronization network link under the offline synchronization task, select the data source and data destination of the offline synchronization task, and the resource group used to execute the synchronization task, test connectivity, edit the script, and configure the offline synchronization task; the production environment operation and maintenance management module is used to import the offline synchronization task into the production environment operation and maintenance center, and the production environment operation and maintenance center calls the large model to configure the dirty data processing node; the intelligent monitoring module is used to perform intelligent monitoring of the offline data development task log according to the dirty data processing node and the offline synchronization task, and control the start and stop of the synchronization task of the offline data development task log;

[0076] The output end of the data integration directory module is connected to the input end of the offline synchronization module; the output end of the offline synchronization module is connected to the input end of the production environment operation and maintenance management module; the output end of the production environment operation and maintenance management module is connected to the input end of the intelligent monitoring module.

[0077] The data integration directory module includes a software development business unit and an offline synchronization task construction unit;

[0078] The software development business unit is used to record the software development business process name and form a software development business list; the offline synchronization task construction unit is used to drag the offline synchronization node and construct the offline synchronization task;

[0079] The output end of the software development business unit is connected to the input end of the offline synchronization task construction unit.

[0080] The offline synchronization module includes an offline synchronization test unit and an offline synchronization configuration unit;

[0081] The offline synchronization test unit is used to configure the synchronization network link under the offline synchronization task, select the data source and data destination of the offline synchronization task, and the resource group used to execute the synchronization task, and test the connectivity; the offline synchronization configuration unit is used to edit the script, and configure the relevant attributes of the automatic scheduling of the task under the periodically scheduled offline synchronization task, and the relevant attributes include: configuration node scheduling attributes, configuration time attributes, and configuration resource attributes;

[0082] The output end of the offline synchronization test unit is connected to the input end of the offline synchronization configuration unit.

[0083] The production environment operation and maintenance management module includes a production environment operation and maintenance center and a large model analysis unit;

[0084] The production environment operation and maintenance center is used to import offline synchronization tasks and call large model processing; the large model analysis unit forms large models under different resource groups based on sample data;

[0085] The output end of the large model analysis unit is connected to the input end of the production environment operation and maintenance center.

[0086] The intelligent monitoring module includes an intelligent monitoring unit and an automatic start-stop control unit;

[0087] The intelligent monitoring unit is used to perform intelligent monitoring of the offline data development task log according to the dirty data processing node and the offline synchronization task, and control the start and stop of the synchronization task of the offline data development task log; the automatic start and stop unit is used to obtain the offline synchronization task and the number of configured dirty data processing nodes under the offline synchronization task, and perform intelligent monitoring of the offline data development task log. If the number of dirty data generated within the period under the configured time attribute is lower than or equal to the number of configured dirty data processing nodes, the synchronization task of the offline data development task log is performed normally; if the number of dirty data generated within the period under the configured time attribute is higher than the number of configured dirty data processing nodes, the synchronization task of the offline data development task log is stopped, the synchronized offline task data is retained, and an error is reported on the administrator port at the same time.

[0088] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0089] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent monitoring method for offline data development task log based on a large model, characterized by: The method comprises the following steps: S1. Obtain the name of the software development business process, drag and drop the offline synchronization node in the data integration directory module, and build an offline synchronization task; S2. Under the offline synchronization task, configure the synchronization network link, select the data source and data destination of the offline synchronization task, and the resource group used to execute the synchronization task, test connectivity, edit the script, and configure the offline synchronization task; S3. Import the offline synchronization task into the production environment operation and maintenance center, and the production environment operation and maintenance center calls the large model to configure the dirty data processing node; S4. Based on the dirty data processing node and the offline synchronization task, the offline data development task log is intelligently monitored to control the start and stop of the synchronization task of the offline data development task log.

2. According to claim 1, a large model-based offline data development task log intelligent monitoring method is characterized by: In step S2, it also includes: Select the switch network segment bound to the resource group, the EIP of the old resource group itself, or the EIP of the new universal resource group bound to the VPC configuration, and add them to the whitelist of data sources; Build a data association between the data source and DataWorks and add a DataWorks data source to test connectivity. Specifically, the following steps are required: In the Data Integration Catalog module, log in to the DataWorks console, select Data Integration in the navigation bar, select the corresponding workspace, enter Data Integration, select Data Source in the navigation bar, and then add a data source in the data source list. Set the newly added data source as the current data source and configure related connection parameters to test connectivity. The configuration offline synchronization task also includes: The periodically scheduled offline synchronization task needs to configure the related attributes when the task is automatically scheduled, and the related attributes include: configuring node scheduling attributes, configuring time attributes, and configuring resource attributes; The configuration node scheduling attributes include: assigning scheduling parameters to custom variables, and supporting the assignment of constants; The configuration time attributes include: a periodic scheduling method for defining tasks in a production environment; The configuration resource attributes include: in a periodically scheduled offline synchronization task scenario, defining a scheduling resource group used when executing resources for a data integration task.

3. According to claim 2, a large model-based offline data development task log intelligent monitoring method is characterized by: In step S3, it also includes: Based on different resource groups that perform synchronization tasks, large models under different resource groups are constructed respectively. The method for constructing a large model under any resource group includes: Construct a sample data set, select the original data under the same resource group, the original data refers to the manual setting data of the user administrator, and form a specific data set ,in Refers to the offline synchronization task feature group, specifically including the total number of offline synchronization tasks and the number of associations between offline synchronization tasks. The association refers to the data interaction between any two offline synchronization tasks, which is defined as the number of associations at one time; Refers to configuring the number of dirty data processing nodes; using a fixed weight format Assign values, including: in, Refers to the weight set by the system; , Represents the normalized data of the total number of offline synchronization tasks and the number of associations between offline synchronization tasks; A frequency histogram is constructed based on the sample data set. The area of ​​the rectangle in the frequency histogram is defined as the interval frequency, and the height of the rectangle is the average frequency density of the interval. Then: in, Refers to sample data i in the sample data set; Refers to the probability density function based on a sample data set; Refers to the cumulative distribution function based on the sample data set; h represents the bandwidth of the kernel density estimate, which is a system setting; , Respectively represent , The cumulative distribution function value under , that is, for continuous functions, all values ​​less than or equal to The value of , and its probability of occurrence is The cumulative distribution function value under ; all values ​​less than or equal to The value of , and its probability of occurrence is The cumulative distribution function value under ; In a specific data set In the two-dimensional scenario, for any new data x, find its closest range in the sample data set ; Form the probability density function calculation for any new data x: Among them, s represents the sequence number; n represents the total number of samples in the sample data set; Represents the uniform kernel function; Represents x and distance; According to the total number of offline synchronization tasks and the number of associations between offline synchronization tasks monitored by the system, a specific new data set is formed, thereby outputting the number of configured dirty data processing nodes under the specific new data set.

4. According to claim 3, a large model-based offline data development task log intelligent monitoring method is characterized by: Also includes: Obtain offline synchronization tasks and the number of dirty data processing nodes configured under the offline synchronization tasks, and perform intelligent monitoring of offline data development task logs. If the number of dirty data generated within the period under the configured time attribute is lower than or equal to the number of dirty data processing nodes configured, the synchronization task of the offline data development task log is performed normally; if the number of dirty data generated within the period under the configured time attribute is higher than the number of dirty data processing nodes configured, the synchronization task of the offline data development task log is stopped, the synchronized offline task data is retained, and an error is reported on the administrator port at the same time.

5. An offline data development task log intelligent monitoring system based on a large model, characterized by: The system includes: data integration directory module, offline synchronization module, production environment operation and maintenance management module and intelligent monitoring module; The data integration directory module is used to obtain the name of the software development business process, drag the offline synchronization node in the data integration directory module, and build an offline synchronization task; the offline synchronization module is used to configure the synchronization network link under the offline synchronization task, select the data source and data destination of the offline synchronization task, and the resource group used to execute the synchronization task, test connectivity, edit the script, and configure the offline synchronization task; the production environment operation and maintenance management module is used to import the offline synchronization task into the production environment operation and maintenance center, and the production environment operation and maintenance center calls the large model to configure the dirty data processing node; the intelligent monitoring module is used to perform intelligent monitoring of the offline data development task log according to the dirty data processing node and the offline synchronization task, and control the start and stop of the synchronization task of the offline data development task log; The output end of the data integration directory module is connected to the input end of the offline synchronization module; the output end of the offline synchronization module is connected to the input end of the production environment operation and maintenance management module; the output end of the production environment operation and maintenance management module is connected to the input end of the intelligent monitoring module.

6. According to claim 5, a large model-based offline data development task log intelligent monitoring system is characterized by: The data integration directory module includes a software development business unit and an offline synchronization task construction unit; The software development business unit is used to record the software development business process name and form a software development business list; the offline synchronization task construction unit is used to drag the offline synchronization node and construct the offline synchronization task; The output end of the software development business unit is connected to the input end of the offline synchronization task construction unit.

7. According to claim 5, a large model-based offline data development task log intelligent monitoring system is characterized by: The offline synchronization module includes an offline synchronization test unit and an offline synchronization configuration unit; The offline synchronization test unit is used to configure the synchronization network link under the offline synchronization task, select the data source and data destination of the offline synchronization task, and the resource group used to execute the synchronization task, and test the connectivity; the offline synchronization configuration unit is used to edit the script, and configure the relevant attributes of the automatic scheduling of the task under the periodically scheduled offline synchronization task, and the relevant attributes include: configuration node scheduling attributes, configuration time attributes, and configuration resource attributes; The output end of the offline synchronization test unit is connected to the input end of the offline synchronization configuration unit.

8. According to claim 5, a large model-based offline data development task log intelligent monitoring system is characterized by: The production environment operation and maintenance management module includes a production environment operation and maintenance center and a large model analysis unit; The production environment operation and maintenance center is used to import offline synchronization tasks and call large model processing; the large model analysis unit forms large models under different resource groups based on sample data; The output end of the large model analysis unit is connected to the input end of the production environment operation and maintenance center.

9. The large model-based offline data development task log intelligent monitoring system according to claim 5 is characterized by: The intelligent monitoring module includes an intelligent monitoring unit and an automatic start-stop control unit; The intelligent monitoring unit is used to perform intelligent monitoring of the offline data development task log according to the dirty data processing node and the offline synchronization task, and control the start and stop of the synchronization task of the offline data development task log; the automatic start and stop unit is used to obtain the offline synchronization task and the number of configured dirty data processing nodes under the offline synchronization task, and perform intelligent monitoring of the offline data development task log. If the number of dirty data generated within the period under the configured time attribute is lower than or equal to the number of configured dirty data processing nodes, the synchronization task of the offline data development task log is performed normally; if the number of dirty data generated within the period under the configured time attribute is higher than the number of configured dirty data processing nodes, the synchronization task of the offline data development task log is stopped, the synchronized offline task data is retained, and an error is reported on the administrator port at the same time.

Citation Information

Patent Citations

  • Fine adjustment method and device for large model in banking business, equipment and storage medium

    CN116579402A

  • Data management platform taking metadata as core and implementation method

    CN116910078A

  • Rapid large model reasoning service method and system suitable for consumer graphics card

    CN117689025A

  • Method of identifying optical axis of laser machining apparatus

    KR102740085B1