Data access and task processing control method and system for large model platform

Abstract

The invention discloses a data access and task processing control method and system for a large model platform, and the method comprises the steps: monitoring an existing task processing record of the large model platform, and obtaining an abnormal processing event of the large model platform for existing task data; a task data mode for processing the abnormal event is traced, and incompatible data attributes of the large model platform are determined, so that local data of a user side is preprocessed, and a data packet queue is generated; adjusting the access strategy of the data packet queue according to the resource use condition of the large model platform; and according to task process planning and resource configuration actual conditions of the large model platform, adjusting a matching import state of the data packet queue during task process execution, and changing an existing state of the data packet in the large model platform. By analyzing task processing historical records of the platform, incompatible data attributes of the platform are calibrated, so that a data packet queue access strategy is adjusted, the data state in the platform is adjusted and changed, and the operation efficiency of a large model platform and the accuracy of a large model output result are improved.

Description

Technical Field

[0001] This invention relates to the field of large model data processing technology, and in particular to a data access and task processing control method and system for large model platforms. Background Technology

[0002] Large-scale model platforms, as business platforms for artificial intelligence, are widely used in complex business processing across various scenarios. In practical applications, training and inputting data into these platforms enables rapid and accurate results. However, large-scale models have limitations in knowledge learning, requiring fine-tuning for different scenarios to achieve the desired performance. Considering the complexity of these platforms and the vast number and variety of data types involved, it's crucial to ensure that input data accurately corresponds to the task processes within the large-scale model. Furthermore, the platform's limited memory and computing resources restrict its processing speed across different task processes. Excessive data access rates during task execution can lead to process lag, impacting the overall reliability of the platform and reducing the accuracy of the results. Therefore, understanding the platform's compatibility with different data types and adjusting data access settings is essential for improving its operational efficiency and the accuracy of its output. Summary of the Invention

[0003] Considering the limitations of large model platforms in terms of input data modality, only compatible with specific modal data, and the fact that the efficiency of large model platform task execution is affected by its own computing resources, inputting a large amount of data during periods of low task execution speed can lead to platform lag and other abnormalities, affecting the overall operational reliability of the large model platform and reducing the accuracy of model processing results. In view of the above problems, this invention is proposed to provide a data access and task processing control method for large model platforms that overcomes or at least partially solves the above problems, including:

[0004] Monitor the existing task processing records of the large model platform to obtain abnormal events in the processing of existing task data by the large model platform; trace the task data modality of the abnormal events to determine the incompatible data attributes of the large model platform.

[0005] Based on the incompatible data attributes, the local data on the user end is preprocessed to generate a data packet queue; based on the actual resource usage of the large model platform, the access strategy of the data packet queue is adjusted.

[0006] Based on the task process planning and resource configuration of the large model platform, adjust the matching and import status of the data packet queue during task execution; based on the execution result of the task process, change the existence status of the data packets on the large model platform.

[0007] Optionally, monitor existing task processing records on the large model platform to obtain abnormal events in the processing of existing task data by the large model platform; trace the task data modality of the abnormal processing events to determine the incompatible data attributes of the large model platform, including:

[0008] Based on the task tags generated locally by the large model platform, the working logs of the large model platform are dynamically tracked to obtain existing task processing records; wherein the existing task processing records include records of changes in computing load during the processing of existing tasks.

[0009] Extract the stage where the computational load exceeds the limit from the existing task processing records, and determine whether a task data processing bottleneck or infinite loop has occurred based on the processing progress of the existing task data corresponding to the stage; if yes, it is determined that an abnormal event in the processing of existing task data has occurred in the stage; if no, it is determined that no abnormal event in the processing of existing task data has occurred in the stage.

[0010] The duration of the abnormal processing state of each content modality sub-data and each format modality sub-data under the task data for handling abnormal events is traced and analyzed; based on the duration of the abnormal processing state, the incompatible data attributes of the large model platform are determined; wherein the incompatible data attributes include the content modality attributes and / or format modality attributes of incompatible data.

[0011] Optionally, based on the incompatible data attributes, the local data on the user end is preprocessed to generate a data packet queue; the access strategy of the data packet queue is adjusted according to the actual resource usage of the large model platform, including:

[0012] Based on the incompatible data attributes, a first attribute label is generated; the local data on the user's end is identified and distinguished to generate a second attribute label; the first attribute label and the second attribute label are compared to identify the incompatible data portion of the local data.

[0013] After performing incompatible data content replacement and / or incompatible data format conversion preprocessing on the incompatible data portion, the local data is divided into data packet queues;

[0014] Based on the actual memory resource usage of the large model platform, the distribution changes of available storage areas within the large model platform are predicted, thereby adjusting the access strategy of the data packet queue; wherein the access strategy includes the compression and storage strategy of the data packet queue.

[0015] Optionally, based on the task process planning and resource configuration of the large model platform, the matching and import status of the data packet queue during task execution is adjusted; based on the execution result of the task process, the existence status of the data packets on the large model platform is changed, including:

[0016] By comparing the process content included in the task process planning of the large model platform with the data packet queue, the correspondence between each task process and each data packet is determined;

[0017] Based on the actual computing resource configuration of the large model platform, estimate the amount of computing resources that each task process is allowed to call;

[0018] Based on the correspondence and the amount of computing resources, adjust the matching import traffic rate status of the data packet queue during task process execution;

[0019] Based on the execution results of the task process, distinguish between successfully completed and unsuccessfully completed task processes, and thereby change the existence status of the data packets corresponding to the successfully completed and unsuccessfully completed task processes on the large model platform.

[0020] As one aspect of the present invention, embodiments of the present invention also provide a data access and task processing control system for a large model platform, including:

[0021] The abnormal event determination module is used to monitor the existing task processing records of the large model platform and obtain the abnormal events in the processing of existing task data by the large model platform.

[0022] The data tracing module is used to trace the task data modality of the abnormal event processing and determine the incompatible data attributes of the large model platform;

[0023] The queue generation module is used to preprocess the local data on the user terminal according to the incompatible data attributes and generate a data packet queue.

[0024] The access adjustment module is used to adjust the access strategy of the data packet queue according to the actual resource usage of the large model platform.

[0025] The import adjustment module is used to adjust the matching import status of the data packet queue during task process execution based on the task process planning and resource configuration of the large model platform.

[0026] The data modification module is used to change the existence status of data packets on the large model platform according to the execution results of the task process.

[0027] Optionally, the abnormal event determination module is used to monitor the existing task processing records of the large model platform to obtain abnormal events in the processing of existing task data by the large model platform, including:

[0028] Based on the task tags generated locally by the large model platform, the working logs of the large model platform are dynamically tracked to obtain existing task processing records; wherein the existing task processing records include records of changes in computing load during the processing of existing tasks.

[0029] Extract the stage where the computational load exceeds the limit from the existing task processing records, and determine whether a task data processing bottleneck or infinite loop has occurred based on the processing progress of the existing task data corresponding to the stage; if yes, it is determined that an abnormal event in the processing of existing task data has occurred in the stage; if no, it is determined that no abnormal event in the processing of existing task data has occurred in the stage.

[0030] The data tracing module is used to trace the task data modality for handling abnormal events and determine the incompatible data attributes of the large model platform, including:

[0031] The duration of the abnormal processing state of each content modality sub-data and each format modality sub-data under the task data for handling abnormal events is traced and analyzed; based on the duration of the abnormal processing state, the incompatible data attributes of the large model platform are determined; wherein the incompatible data attributes include the content modality attributes and / or format modality attributes of incompatible data.

[0032] Optionally, the queue generation module is used to preprocess the local data on the user terminal according to the incompatible data attributes to generate a data packet queue, including:

[0033] Based on the incompatible data attributes, a first attribute label is generated; the local data on the user's end is identified and distinguished to generate a second attribute label; the first attribute label and the second attribute label are compared to identify the incompatible data portion of the local data.

[0034] After performing incompatible data content replacement and / or incompatible data format conversion preprocessing on the incompatible data portion, the local data is divided into data packet queues;

[0035] The access adjustment module is used to adjust the access strategy of the data packet queue according to the actual resource usage of the large model platform, including:

[0036] Based on the actual memory resource usage of the large model platform, the distribution changes of available storage areas within the large model platform are predicted, thereby adjusting the access strategy of the data packet queue; wherein the access strategy includes the compression and storage strategy of the data packet queue.

[0037] Optionally, the import adjustment module is used to adjust the matching import status of the data packet queue during task execution based on the task process planning and resource configuration of the large model platform, including:

[0038] By comparing the process content included in the task process planning of the large model platform with the data packet queue, the correspondence between each task process and each data packet is determined;

[0039] Based on the actual computing resource configuration of the large model platform, estimate the amount of computing resources that each task process is allowed to call;

[0040] Based on the correspondence and the amount of computing resources, adjust the matching import traffic rate status of the data packet queue during task process execution;

[0041] The data modification module is used to change the existence status of data packets on the large model platform according to the execution result of the task process, including:

[0042] Based on the execution results of the task process, distinguish between successfully completed and unsuccessfully completed task processes, and thereby change the existence status of the data packets corresponding to the successfully completed and unsuccessfully completed task processes on the large model platform.

[0043] The beneficial effects of the above-mentioned technical solutions provided in the embodiments of the present invention include at least the following:

[0044] This invention provides a data access and task processing control method for a large-scale model platform, and a system that monitors existing task processing records of the large-scale model platform to obtain abnormal events in the processing of existing task data. It traces the task data modalities that handle these abnormal events to determine incompatible data attributes of the large-scale model platform. Based on these incompatible data attributes, it preprocesses local data on the user end to generate a data packet queue. It adjusts the data packet queue access strategy based on the actual resource usage of the large-scale model platform. It adjusts the matching and import status of the data packet queue during task execution based on the task process planning and resource configuration of the large-scale model platform. Finally, it changes the existence status of the data packets on the large-scale model platform based on the execution results of the task process. By analyzing the platform's task processing history, incompatible data attributes are identified, thereby adjusting the data packet queue access strategy. Furthermore, by combining task process planning and resource configuration, the internal data status of the platform is adjusted and changed, improving the operating efficiency of the large-scale model platform and the accuracy of its output results.

[0045] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0046] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0047] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0048] Figure 1 This is a flowchart illustrating the data access and task processing control method for a large model platform provided in this embodiment of the invention.

[0049] Figure 2 This is a schematic diagram of the data access and task processing control system of the large model platform provided in this embodiment of the invention. Detailed Implementation

[0050] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0051] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," "outer," "far," "near," "front," and "rear," etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0052] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0053] Please see Figure 1As shown in this application, an embodiment of the data access and task processing control method for a large model platform is provided. This data access and task processing control method for a large model platform includes:

[0054] Monitor the existing task processing records of the large model platform to obtain abnormal events in the processing of existing task data by the large model platform; trace the task data modality that handles abnormal events to determine the incompatible data attributes of the large model platform.

[0055] Based on incompatible data attributes, preprocess the local data on the user end to generate a data packet queue; adjust the access strategy of the data packet queue according to the actual resource usage of the large model platform.

[0056] Based on the task process planning and resource configuration of the large model platform, adjust the matching and import status of the data packet queue during task execution; based on the execution results of the task process, change the existence status of the data packets on the large model platform.

[0057] The beneficial effects of the above embodiments are that the data access and task processing control method of the large model platform analyzes the platform's task processing history, identifies incompatible data attributes, and adjusts the data packet queue access strategy accordingly; it also combines task process planning and resource allocation to adjust and change the data state inside the platform, thereby improving the operating efficiency of the large model platform and the accuracy of the large model output results.

[0058] In another embodiment, the existing task processing records of the large model platform are monitored to obtain abnormal events in the processing of existing task data by the large model platform; the task data modality of the abnormal events is traced to determine the incompatible data attributes of the large model platform, including:

[0059] Based on the task tags generated locally on the large model platform, the working logs of the large model platform are dynamically tracked to obtain existing task processing records; these records include records of changes in computational load during the processing of existing tasks.

[0060] Extract the stage where the computational load exceeds the limit from the existing task processing records. Based on the processing progress of the existing task data corresponding to the stage, determine whether there is a task data processing bottleneck or infinite loop. If so, it is determined that an abnormal event has occurred in the processing of existing task data in the stage. If not, it is determined that no abnormal event has occurred in the processing of existing task data in the stage.

[0061] The duration of the abnormal state in each content modality sub-data and each format modality sub-data under the task data for handling abnormal events is traced and analyzed. Based on the duration of the abnormal state, the incompatible data attributes of the large model platform are determined. The incompatible data attributes include the content modality attributes and / or format modality attributes of the incompatible data.

[0062] The beneficial effects of the above embodiments are that the large model platform loads specific types of large models, which have certain limitations for the data to be processed. These limitations are mainly reflected in the data content and data format. When the data to be processed is compatible in content and format, the large model can complete the data processing with relatively small computing resources; when the data to be processed is incompatible in content or format, the large model requires more computing resources to process it, and abnormal situations such as lag or infinite loops may occur during the processing. Under normal circumstances, in order to ensure that the large model platform processes data efficiently and smoothly and reduce the waste of computing resources, it is necessary to identify and transform the data to meet the compatibility characteristics of the large model platform. First, based on the task tags generated locally by the large model platform, the working log of the large model platform is dynamically tracked to obtain the record of the changes in the computational load during the processing of existing tasks (i.e., tasks already loaded by the large model platform). The aforementioned records of changes in computational load are correlated with whether the large model platform processes data that is incompatible in content or format. Therefore, the stages where the computational load exceeds the limit (i.e., stages where the computational load exceeds the preset load threshold) are extracted from the existing task processing records. Based on the processing progress of the existing task data corresponding to the aforementioned stages, if the processing progress is less than the preset progress threshold, it is determined that a task data processing bottleneck or infinite loop has occurred, thus confirming that an abnormal event has occurred in the processing of existing task data in the stage; otherwise, it is determined that no task data processing bottleneck or infinite loop has occurred, thus confirming that no abnormal event has occurred in the processing of existing task data in the stage.

[0063] Furthermore, the duration of the abnormal state processing for each content modality sub-data and each format modality sub-data under the corresponding task data during the abnormal event processing is determined. If the duration of the abnormal state processing exceeds a preset time threshold, the corresponding content modality and / or format modality are determined to be content modality attributes and / or format modality attributes of data that are incompatible with the large model platform. This provides a basis for subsequently preprocessing the user's local data into data compatible with the large model platform.

[0064] In one embodiment, the aforementioned "monitoring existing task processing records of the large model platform to obtain abnormal events in the processing of existing task data by the large model platform; tracing the task data modality of the abnormal events to determine the incompatible data attributes of the large model platform" can also be implemented as follows:

[0065] The step of "monitoring the existing task processing records of the large model platform and obtaining abnormal events in the processing of existing task data by the large model platform" includes steps S21-23:

[0066] Step 21: Collect the average computing load of the platform and the amount of data actually processed by the platform in each sampling period according to the preset sampling period.

[0067] Specifically, the duration of the sampling period is set. The system periodically collects the following data according to this sampling period: the average computing load of the platform within the nth sampling period. Collect the actual amount of data processed by the platform during the current sampling period. ;in The sampling period number is... .

[0068] in, The sampling period duration, with a range of values, for example: The time in seconds can be set through the system monitoring and configuration module; For the first The average computing load of the platform within each sampling period can be expressed as a percentage of CPU / GPU utilization, with a value range of [value range missing]. This information is obtained through the operating system performance monitoring interface or hardware monitoring sensors. For the first The actual amount of data successfully processed by the platform within a sampling period, measured in rows or bytes, and in non-negative real numbers, is obtained by parsing the task manager logs of the large model platform. It is the sum of the data amounts of all successfully processed tasks within that period.

[0069] Step 22: For each sampling period, determine the anomaly index corresponding to that sampling period based on the platform's average computing load, the actual amount of data processed by the platform, the duration of each sampling period, the expected processing time corresponding to the actual amount of data processed by the platform, the preset normal platform load threshold, and the preset maximum platform load threshold.

[0070] Specifically, the first step can be determined according to the following formula (1). Actual processing rate per sampling period :

[0071] (1)

[0072] Then, determine the first according to the following formula (2). Anomaly index for each sampling period: (2)

[0073] in, For the first The expected processing rate of the task processed per sampling period can be expressed, for example, by the formula... Calculated, where, For the first Total data volume of the task in each sampling period; The expected total time for the task's total data volume is obtained from the database of average time for similar historical tasks. The preset normal platform load threshold is expressed as a utilization percentage, with a value range of, for example, . The determination was made through system stability testing and empirical configuration. This is the preset maximum platform load threshold, typically 100%, determined by the hardware specifications. For the weight parameters, satisfying The range of values ​​is The settings can be dynamically adjusted through the task type configuration file; for example, the settings can be increased for computationally intensive tasks. For I / O-intensive tasks, the setting can be increased. ; The smoothing constant has a value of [value]. Fixed parameters to prevent the denominator from being zero; No. The anomaly index for each sampling period; the higher the value, the greater the degree of anomaly.

[0074] Among them, in determining When matching tasks, historical similar tasks can be matched based on the following characteristics: task type (such as text generation, image recognition, speech processing), data size classification (small: <1GB, medium: 1-10GB, large: >10GB), model type (such as GPT series, BERT series, visual Transformer, etc.).

[0075] Step 23: When the abnormal index of K consecutive sampling periods is equal to or greater than the preset abnormal index threshold, it is determined that an abnormal processing event has occurred.

[0076] Among them, K ,in, The minimum duration of the anomaly can be preset. If the abnormal index is abnormal in each sampling period, an abnormal processing event is determined to have occurred. This is the floor function.

[0077] Accordingly, the step of "tracing the task data modality for handling abnormal events and determining the incompatible data attributes of the large model platform" includes the following steps:

[0078] Step 31: For each abnormal event: Determine the sampling period range covered by the abnormal event, extract all modal labels of the platform-processed data in each sampling period within the sampling period range, merge all modal labels of all sampling periods to form the data modality set of the abnormal event; determine the time decay weight of the abnormal event based on the current time and the end time of the abnormal event.

[0079] The data modality set includes a set of tags for content modalities (such as text, images, audio, and video) and format modalities (such as JSON, XML, CSV, and Parquet), which is obtained by parsing the data metadata of each sampling period in the platform task log.

[0080] The time decay weight for the i-th abnormal event processing can be determined according to the following formula (3). :

[0081] (3)

[0082] in, Let be the end time of the i-th exception handling; Indicates the current time. This is a preset decay time constant, in units of time, with a range of values, for example, [range missing]. Seconds; the more recent the anomaly. The closer it is to 1.

[0083] Step 32: Determine the anomaly correlation degree of each data modality based on the maximum anomaly index, data modality set, and time decay weight of each of the N anomaly handling events that occurred within the preset historical time period.

[0084] The abnormal correlation degree of each data modality can be determined according to the following formula (4):

[0085] (4)

[0086] in, This represents the degree of abnormal correlation of the k-th data modality; the larger the value, the stronger the correlation between the modality and the abnormal event. This is an indicator function that indicates when the k-th data modality occurs during the i-th processing exception event. Data modal set When the condition is met, the function value is 1; otherwise, it is 0. This represents the maximum value of the anomaly index for the i-th anomaly event, which is the maximum value of the anomaly index corresponding to all sampling periods in the i-th anomaly event processing. This is a smoothing constant, a preset value, with values ​​such as 1. Fixed parameters to prevent the denominator from being zero.

[0087] Step 33: Determine whether each data modality is an incompatible data attribute based on its abnormal correlation degree.

[0088] Step 33 can be implemented as the following sub-steps:

[0089] Step 331: Determine the adaptive judgment threshold based on the total number of data modes and the abnormal correlation degree of each data mode;

[0090] Specifically, the adaptive decision threshold can be determined according to the following formula (5):

[0091] (5)

[0092] in, The total number of data modalities is a positive integer, obtained by analyzing all modalities that have appeared on the statistical platform. This is a preset sensitivity parameter, dimensionless, with a preset value and a range of values, for example: The tolerance for false alarms and false negatives is determined by configuring the system.

[0093] Step 332: Determine whether each data modality is an incompatible data attribute based on the relationship between the abnormal correlation degree of each data modality and the adaptive judgment threshold.

[0094] Specifically, when the abnormal correlation of a data modality is greater than the adaptive judgment threshold, the data modality is determined to be an incompatible data attribute.

[0095] The beneficial effects of the above technical solution are as follows: It provides a platform compatibility diagnosis method based on quantitative analysis and adaptive learning. By introducing a sampling period to monitor the computational load and processing rate in real time, and constructing an anomaly exponential function to dynamically identify and handle abnormal events; further, by utilizing time decay weights and modal correlation calculations, it achieves automatic tracing and accurate determination of platform-incompatible data attributes. This technical solution, through personalized parameter configuration and adaptive adjustment of statistical thresholds, can optimize the platform's data compatibility identification capability, not only improving the accuracy and timeliness of anomaly detection, but also providing a reliable basis for subsequent data preprocessing, thereby improving the overall operating efficiency, processing reliability, and output accuracy of the large model platform.

[0096] In another embodiment, based on incompatible data attributes, the user's local data is preprocessed to generate a data packet queue; the access strategy of the data packet queue is adjusted according to the actual resource usage of the large model platform, including:

[0097] Based on the incompatible data attributes, generate a first attribute label; perform attribute identification and differentiation on the user's local data to generate a second attribute label; compare the first attribute label and the second attribute label to identify the incompatible data portion of the local data;

[0098] After preprocessing the incompatible data by replacing the incompatible data content and / or converting the incompatible data format, the local data is divided into data packet queues;

[0099] Based on the actual memory resource usage of the large model platform, predict the changes in the distribution of available storage areas within the large model platform, and adjust the access strategy of the data packet queue accordingly; the access strategy includes the compression and storage strategies of the data packet queue.

[0100] The beneficial effects of the above embodiments are as follows: In actual operation, a first attribute label in vector form is generated based on the content modal attributes and / or format modal attributes of the incompatible data attributes; multi-dimensional attribute identification and differentiation are performed on the local data of the user terminal to generate a second attribute label in vector form; the first attribute label and the second attribute label are compared to obtain their similarity, which is used to identify the incompatible data part of the local data. It can be understood that the similarity between the second attribute label corresponding to the incompatible data part and the first attribute label exceeds a preset similarity threshold. Then, the incompatible data part is preprocessed by incompatible data content replacement and / or incompatible data format conversion to obtain converted data that is compatible with the large model platform in terms of content and format. The converted data and the compatible data part in the local data are reintegrated, segmented and packaged to obtain a data packet queue. In this way, the data in each data packet in the data packet queue is compatible with the large model platform in terms of content and format.

[0101] Furthermore, based on the actual memory resource usage of the large model platform, the changes in the distribution of available storage areas within the platform are predicted. These changes refer to the shifts in the location of storage areas that become idle within the platform. The compression and storage strategies for data packet queues are then adjusted based on these changes to ensure that data packets transmitted to the large model platform are accurately stored in idle storage areas, thereby improving the reliability of data storage within the platform.

[0102] In another embodiment, the matching and import status of the data packet queue during task execution is adjusted according to the task process planning and resource configuration of the large model platform; the existence status of the data packets on the large model platform is changed according to the execution result of the task process, including:

[0103] By comparing the process content and data packet queues included in the task process planning of the large model platform, the correspondence between each task process and each data packet is determined;

[0104] Based on the actual computing resource configuration of the large model platform, estimate the amount of computing resources that each task process is allowed to call;

[0105] Based on the correspondence and the amount of computing resources, adjust the matching import traffic rate status of the data packet queue during task execution;

[0106] Based on the execution results of the task process, distinguish between successfully completed and unsuccessfully completed task processes, and thereby change the existence status of the data packets corresponding to the successfully completed and unsuccessfully completed task processes on the large model platform.

[0107] The beneficial effects of the above embodiments are that each task process processed within the large model platform uniquely corresponds to corresponding data. Only when a task process is imported with matching data can its correct execution be guaranteed. To improve the task execution efficiency and reliability of the large model platform, the process content and data packet queue included in the task process plan of the large model platform are compared to determine the correspondence between each task process and each data packet. This correspondence reflects a one-to-one relationship between task processes and data packets. Then, based on the actual computing resource configuration of the large model platform, the amount of computing resources that each task process is allowed to call is estimated. Combined with the above correspondence, the matching import traffic rate of the data packet queue during task process execution is adjusted, ensuring that each task process receives a data packet supply with a matching import traffic rate during execution. This avoids the data packet supply traffic rate being too high or too low, which could affect the execution stability and efficiency of the task process.

[0108] Furthermore, based on the execution results of the task process, the system distinguishes between successfully completed and unsuccessfully completed task processes, thereby changing the existence status of the data packets corresponding to each process on the large model platform. For example, it can directly delete the data packets corresponding to successfully completed task processes from the large model platform, while retaining and reusing the data packets corresponding to unsuccessfully completed task processes to avoid inputting data packets from the outside to support the re-execution of the aforementioned unsuccessfully completed task processes, thus reducing the data interaction load on the large model platform.

[0109] Please see Figure 2 As shown in one embodiment of this application, a data access and task processing control system for a large-scale model platform is provided. This data access and task processing control system for the large-scale model platform includes:

[0110] The exception event determination module is used to monitor the existing task processing records of the large model platform and obtain the exception events of the large model platform in processing existing task data.

[0111] The data tracing module is used to trace the task data modalities for handling abnormal events and to determine incompatible data attributes of the large model platform;

[0112] The queue generation module is used to preprocess local data on the user end based on incompatible data attributes and generate data packet queues.

[0113] The access adjustment module is used to adjust the access strategy of the data packet queue based on the actual resource usage of the large model platform.

[0114] The import adjustment module is used to adjust the matching import status of the data packet queue during task execution based on the task process planning and resource configuration of the large model platform.

[0115] The data modification module is used to change the existence status of data packets on the large model platform based on the execution results of the task process.

[0116] The beneficial effects of the above embodiments are that the data access and task processing control system of the large model platform analyzes the platform's task processing history, identifies incompatible data attributes, and adjusts the data packet queue access strategy accordingly; it also combines task process planning and resource allocation to adjust and change the data status inside the platform, thereby improving the operating efficiency of the large model platform and the accuracy of the large model output results.

[0117] In another embodiment, the exception event determination module is used to monitor the existing task processing records of the large model platform to obtain exception events in the processing of existing task data by the large model platform, including:

[0118] Based on the task tags generated locally on the large model platform, the working logs of the large model platform are dynamically tracked to obtain existing task processing records; these records include records of changes in computational load during the processing of existing tasks.

[0119] Extract the stage where the computational load exceeds the limit from the existing task processing records. Based on the processing progress of the existing task data corresponding to the stage, determine whether there is a task data processing bottleneck or infinite loop. If so, it is determined that an abnormal event has occurred in the processing of existing task data in the stage. If not, it is determined that no abnormal event has occurred in the processing of existing task data in the stage.

[0120] The data tracing module is used to trace the task data modalities for handling abnormal events and to determine incompatible data attributes of the large model platform, including:

[0121] The duration of the abnormal state in each content modality sub-data and each format modality sub-data under the task data for handling abnormal events is traced and analyzed. Based on the duration of the abnormal state, the incompatible data attributes of the large model platform are determined. The incompatible data attributes include the content modality attributes and / or format modality attributes of the incompatible data.

[0122] In another embodiment, the queue generation module is used to preprocess the user's local data based on incompatible data attributes to generate a data packet queue, including:

[0123] Based on the incompatible data attributes, generate a first attribute label; perform attribute identification and differentiation on the user's local data to generate a second attribute label; compare the first attribute label and the second attribute label to identify the incompatible data portion of the local data;

[0124] After preprocessing the incompatible data by replacing the incompatible data content and / or converting the incompatible data format, the local data is divided into data packet queues;

[0125] The access adjustment module is used to adjust the access strategy of the data packet queue based on the actual resource usage of the large model platform, including:

[0126] Based on the actual memory resource usage of the large model platform, predict the changes in the distribution of available storage areas within the large model platform, and adjust the access strategy of the data packet queue accordingly; the access strategy includes the compression and storage strategies of the data packet queue.

[0127] In another embodiment, the import adjustment module is used to adjust the matching import status of the data packet queue during task execution based on the task process planning and resource configuration of the large model platform, including:

[0128] By comparing the process content and data packet queues included in the task process planning of the large model platform, the correspondence between each task process and each data packet is determined;

[0129] Based on the actual computing resource configuration of the large model platform, estimate the amount of computing resources that each task process is allowed to call;

[0130] Based on the correspondence and the amount of computing resources, adjust the matching import traffic rate status of the data packet queue during task execution;

[0131] The data modification module is used to change the existence status of data packets on the large model platform based on the execution results of the task process, including:

[0132] Based on the execution results of the task process, distinguish between successfully completed and unsuccessfully completed task processes, and thereby change the existence status of the data packets corresponding to the successfully completed and unsuccessfully completed task processes on the large model platform.

[0133] The data access and task processing control system of the large model platform of the present invention has the same operation and effect as the data access and task processing control method of the large model platform described above, and will not be described again here.

[0134] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. This disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims. Thus, if these modifications and variations of the invention fall within the scope of the claims of the invention and their equivalents, the invention is also intended to include these modifications and variations.

Claims

1. A method for data access and task processing control of a large model platform, characterized in that, include: Monitor the existing task processing records of the large model platform to obtain abnormal events in the processing of existing task data by the large model platform; Trace the task data modality of the abnormal event processing to determine the incompatible data attributes of the large model platform; Based on the incompatible data attributes, the local data on the user end is preprocessed to generate a data packet queue; based on the actual resource usage of the large model platform, the access strategy of the data packet queue is adjusted. Based on the task process planning and resource configuration of the large model platform, adjust the matching and import status of the data packet queue during task execution; based on the execution result of the task process, change the existence status of the data packets on the large model platform.

2. The data access and task processing control method for a large model platform as described in claim 1, characterized in that: Monitor the existing task processing records of the large model platform to obtain abnormal events in the processing of existing task data by the large model platform; Tracing the task data modality for handling abnormal events, determining the incompatible data attributes of the large model platform, including: Based on the task tags generated locally on the large model platform, the working logs of the large model platform are dynamically tracked to obtain existing task processing records. The existing task processing records include records of computational load changes during existing task processing; Extract the stage where the computational load exceeds the limit from the existing task processing records, and determine whether a task data processing bottleneck or infinite loop has occurred based on the processing progress of the existing task data corresponding to the stage; if yes, it is determined that an abnormal event in the processing of existing task data has occurred in the stage; if no, it is determined that no abnormal event in the processing of existing task data has occurred in the stage. The duration of the abnormal processing state of each content modality sub-data and each format modality sub-data under the task data for handling abnormal events is traced and analyzed; based on the duration of the abnormal processing state, the incompatible data attributes of the large model platform are determined; wherein the incompatible data attributes include the content modality attributes and / or format modality attributes of incompatible data.

3. The data access and task processing control method for a large model platform as described in claim 1, characterized in that: Based on the incompatible data attributes, the local data on the user end is preprocessed to generate a data packet queue; Based on the actual resource usage of the large model platform, adjust the access strategy of the data packet queue, including: Based on the incompatible data attributes, a first attribute label is generated; the local data on the user's end is identified and distinguished to generate a second attribute label; the first attribute label and the second attribute label are compared to identify the incompatible data portion of the local data. After performing incompatible data content replacement and / or incompatible data format conversion preprocessing on the incompatible data portion, the local data is divided into data packet queues; Based on the actual memory resource usage of the large model platform, the distribution changes of available storage areas within the large model platform are predicted, thereby adjusting the access strategy of the data packet queue; wherein the access strategy includes the compression and storage strategy of the data packet queue.

4. The data access and task processing control method for a large model platform as described in claim 1, characterized in that: Based on the task process planning and resource configuration of the large model platform, adjust the matching and import status of the data packet queue during task process execution; Based on the execution results of the task process, change the existence status of the data packet on the large model platform, including: By comparing the process content included in the task process planning of the large model platform with the data packet queue, the correspondence between each task process and each data packet is determined; Based on the actual computing resource configuration of the large model platform, estimate the amount of computing resources that each task process is allowed to call; Based on the correspondence and the amount of computing resources, adjust the matching import traffic rate status of the data packet queue during task process execution; Based on the execution results of the task process, distinguish between successfully completed and unsuccessfully completed task processes, and thereby change the existence status of the data packets corresponding to the successfully completed and unsuccessfully completed task processes on the large model platform.

5. A data access and task processing control system for a large model platform, characterized in that: include: The abnormal event determination module is used to monitor the existing task processing records of the large model platform and obtain the abnormal events in the processing of existing task data by the large model platform. The data tracing module is used to trace the task data modality of the abnormal event processing and determine the incompatible data attributes of the large model platform; The queue generation module is used to preprocess the local data on the user terminal according to the incompatible data attributes and generate a data packet queue. The access adjustment module is used to adjust the access strategy of the data packet queue according to the actual resource usage of the large model platform. The import adjustment module is used to adjust the matching import status of the data packet queue during task process execution based on the task process planning and resource configuration of the large model platform. The data modification module is used to change the existence status of data packets on the large model platform according to the execution results of the task process.

6. The data access and task processing control system for the large model platform as described in claim 5, characterized in that: The abnormal event determination module is used to monitor the existing task processing records of the large model platform to obtain abnormal events in the processing of existing task data by the large model platform, including: Based on the task tags generated locally by the large model platform, the working logs of the large model platform are dynamically tracked to obtain existing task processing records; wherein the existing task processing records include records of changes in computing load during the processing of existing tasks. Extract the stage where the computational load exceeds the limit from the existing task processing records, and determine whether a task data processing bottleneck or infinite loop has occurred based on the processing progress of the existing task data corresponding to the stage; if yes, it is determined that an abnormal event in the processing of existing task data has occurred in the stage; if no, it is determined that no abnormal event in the processing of existing task data has occurred in the stage. The data tracing module is used to trace the task data modality for handling abnormal events and determine the incompatible data attributes of the large model platform, including: The duration of the abnormal processing state of each content modality sub-data and each format modality sub-data under the task data for handling abnormal events is traced and analyzed; based on the duration of the abnormal processing state, the incompatible data attributes of the large model platform are determined; wherein the incompatible data attributes include the content modality attributes and / or format modality attributes of incompatible data.

7. The data access and task processing control system for the large model platform as described in claim 5, characterized in that: The queue generation module is used to preprocess the local data on the user terminal according to the incompatible data attributes and generate a data packet queue, including: Based on the incompatible data attributes, a first attribute label is generated; the local data on the user's end is identified and distinguished to generate a second attribute label; the first attribute label and the second attribute label are compared to identify the incompatible data portion of the local data. After performing incompatible data content replacement and / or incompatible data format conversion preprocessing on the incompatible data portion, the local data is divided into data packet queues; The access adjustment module is used to adjust the access strategy of the data packet queue according to the actual resource usage of the large model platform, including: Based on the actual memory resource usage of the large model platform, the distribution changes of available storage areas within the large model platform are predicted, thereby adjusting the access strategy of the data packet queue; wherein the access strategy includes the compression and storage strategy of the data packet queue.

8. The data access and task processing control system for the large model platform as described in claim 5, characterized in that: The import adjustment module is used to adjust the matching import status of the data packet queue during task execution based on the task process planning and resource configuration of the large model platform, including: By comparing the process content included in the task process planning of the large model platform with the data packet queue, the correspondence between each task process and each data packet is determined; Based on the actual computing resource configuration of the large model platform, estimate the amount of computing resources that each task process is allowed to call; Based on the correspondence and the amount of computing resources, adjust the matching import traffic rate status of the data packet queue during task process execution; The data modification module is used to change the existence status of data packets on the large model platform according to the execution result of the task process, including: Based on the execution results of the task process, distinguish between successfully completed and unsuccessfully completed task processes, and thereby change the existence status of the data packets corresponding to the successfully completed and unsuccessfully completed task processes on the large model platform.

Images