System and method for operating training data in large model scene

Abstract

The invention provides a system and a method for operating training data in a large model scene, which can improve data processing efficiency and reduce human intervention and error rate. The system for operating the training data in the large model scene comprises a data collection module used for collecting data resources from multi-source channels; the data set management module is in communication connection with the data collection module, and the data set management module is used for registering, inputting, hooking and maintaining the collected data resources; the data auditing module is in communication connection with the data set management module, and the data auditing module is used for performing data auditing on the input data resources; the data auditing comprises data set auditing and process auditing; and the data publishing module is in communication connection with the data auditing module, and the data publishing module is used for publishing the data resources passing the data auditing.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a system and method for operating training data in a large model scenario. Background Art

[0002] With the continuous advancement of artificial intelligence technology and the continuous expansion of its application fields, the demand for data in large model training is increasing, and the requirements for data processing efficiency and quality are also increasing. However, traditional data collection and management methods often have problems such as low efficiency, frequent human intervention, and low degree of automation, which cannot meet the high quality and high efficiency requirements of large model training for data. Summary of the invention

[0003] In order to solve the above technical problems, the present invention is proposed. The embodiments of the present invention provide a system and method for operating training data in a large model scenario, which can improve data processing efficiency and reduce human intervention and error rate.

[0004] According to one aspect of the present invention, there is provided a system for operating training data in a large model scenario, comprising: a data collection module, the data collection module being used to collect data resources from multiple source channels; a data set management module, the data set management module being communicatively connected to the data collection module, the data set management module being used to register, enter, attach and maintain the collected data resources; a data review module, the data review module being communicatively connected to the data set management module, the data review module being used to perform data review on the entered data resources; the data review comprises a data set review and a process review; and a data publishing module, the data publishing module being communicatively connected to the data review module, the data publishing module being used to publish data resources that have passed the data review.

[0005] In one embodiment, the data collection module includes: a transmission unit, which is used to execute a breakpoint resume mechanism when collecting data resources; an encryption unit, which is communicatively connected to the transmission unit and is used to encrypt data resources during the data collection process.

[0006] In one embodiment, the data set management module includes: a registration unit, the registration unit is used to register the collected data resources as a data set; wherein the data set name is the data source, and the data set description includes data set detailed information; a first input unit, the first input unit is used to enter a single data set at a time; wherein a single data set is directly registered on the page; a second input unit, the second input unit is used to enter multiple data sets in words; wherein multiple data sets are filled in a standard Excel file and then uploaded and entered.

[0007] In one embodiment, the data set management module includes: a resource attachment unit, which is used to attach data resources to data file resources or database table resources; wherein the data file resources include at least one of PDF, EXCEL, CSV and JSON, and the database table resources include at least one of MySQL, Oracle and API interface; a verification unit, which is communicatively connected to the resource attachment unit, and is used to perform network verification and authority verification when data is attached; a mapping unit, which is communicatively connected to the resource attachment unit, and is used to perform association mapping of corresponding information items when data is attached.

[0008] In one embodiment, the data set management module also includes: an information change maintenance unit, which is used to synchronize changes when the data set information changes; and a resource change maintenance unit, which is used to update and supplement the data file resources or database table resources attached to the data set when the data set information changes.

[0009] In one embodiment, the data review module includes: a data set review unit, which is used to perform at least one of registration review, change review, offline review and release review; a process review unit, which is used to perform internal department review and company-level review; wherein the review content includes at least one of the standardization of data set registration information, whether the information items are accurate and whether the resources are complete.

[0010] In one embodiment, the data set review unit includes: a timeliness review subunit, which is used to review the time interval of the data set and determine whether the data set within the preset time interval is entered; an accuracy review subunit, which is used to review whether there is source information of the data set and the accuracy of the source information; a uniqueness review subunit, which is used to review the similarity between the entered data set and the existing data set. When the similarity is greater than or equal to the preset similarity, it is determined that the entered data set is duplicated and is not entered; an integrity review subunit, which is used to calculate the data missing rate of the data set. When the data missing rate is less than the preset missing rate, it is entered; a consistency review subunit, which is used to review the consistency of the data in the data set in terms of naming, format, structure and constraint rules; and a compliance review subunit, which is used to confirm that there is no sensitive data or non-compliant data in the included data through keyword retrieval technology.

[0011] In one embodiment, the data publishing module includes: a publishing unit, which is used to provide different publishing methods based on usage requirements; a data service unit, which is used to provide online data interface development and support at least one data service among data interface, data push, data direct connection and file download; a permission management and access control unit, which is communicated with the publishing unit and the data service unit, and the permission management and access control unit is used to manage the usage rights and access rights of publishing and data services.

[0012] In one embodiment, the system for operating training data in a large model scenario also includes: a task scheduling module, which is communicatively connected to the data collection module, the data set management module, the data review module and the data publishing module, and the task scheduling module is used to adjust the task execution frequency and task execution priority based on task requirements; a monitoring module, which is communicatively connected to the task scheduling module, and the monitoring module is used to monitor the task execution status, and when the task execution status is abnormal, an alarm notification is issued.

[0013] According to another aspect of the present invention, a method for operating training data in a large model scenario is provided, which is applied to the system for operating training data in a large model scenario described in any one of the above embodiments. The method for operating training data in a large model scenario includes: collecting data resources from multiple source channels through a data collection module; registering, entering, attaching and maintaining the collected data resources through a data set management module; performing data audit on the entered data resources through a data review module; the data audit includes data set audit and process audit; and publishing the data resources that have passed the data audit through a data publishing module.

[0014] The system and method for operating training data in a large model scenario provided by the present invention constructs a large model training data operation framework, collects large model training data from multiple source channels, and sequentially performs a full-link collection and management process of data set information registration or import, data set attachment to data resources, data set information review, data set information release, data set management, and data resource management, thereby realizing rapid data collection, efficient management, and correct release. Through this framework, the data processing process can be optimized, the efficiency of data collection, processing, and management can be effectively improved, human intervention and error rates can be reduced, and massive, high-quality data support can be provided for large model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a structural diagram of a system for operating training data in a large model scenario provided by an exemplary embodiment of the present invention.

[0017] Figure 2 It is a schematic diagram of a cross-functional process of data operations provided by an exemplary embodiment of the present invention.

[0018] Figure 3 It is a flowchart of a method for operating training data in a large model scenario provided by an exemplary embodiment of the present invention.

[0019] Figure 4 It is a flowchart of an operation mode of operating training data in a large model scenario provided by an exemplary embodiment of the present invention.

[0020] Explanation of the accompanying drawings: 1. System for operating training data in a large model scenario; 11. Data collection module; 12. Data set management module; 13. Data review module; 14. Data publishing module. DETAILED DESCRIPTION

[0021] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0022] The present invention adopts a variety of open source components to achieve efficient data transmission and management. At the same time, the system improves the automatic processing flow through mechanisms such as scheduled tasks and caches, effectively improving data collection and management efficiency. The system aims to provide a comprehensive, efficient and intelligent data collection and management processing solution for large model training work, lay a solid foundation for subsequent collection of massive and high-quality data, and further promote the automation process of large model training data collection work. For example, it combines mainstream open source technologies such as Zookeeper and Kafka, and also realizes the automation and intelligence of data processing through mechanisms such as xxl-job scheduled tasks and redis cache, providing new solutions for data operations in the field of large model training. In addition, the technology is also highly scalable and can adapt to data processing needs of different scales, providing a solid data foundation for the development of the field of artificial intelligence.

[0023] Figure 1 This is a schematic diagram of the structure of a system for operating training data in a large model scenario provided by an exemplary embodiment of the present invention. Figure 1 The system for operating training data in a large model scenario is specifically described. The system 1 for operating training data in a large model scenario includes: a data collection module 11, which is used to collect data resources from multiple source channels; a data set management module 12, which is connected to the data collection module 11 in communication, and is used to register, enter, connect and maintain the collected data resources; a data review module 13, which is connected to the data set management module 12 in communication, and is used to perform data review on the entered data resources; data review includes data set review and process review; a data publishing module 14, which is connected to the data review module 13 in communication, and is used to publish data resources that have passed the data review.

[0024] The data collection module 11 automatically collects the data resources required for large model training from multiple sources by designing efficient data collection strategies. The data collection module 11 is designed to support multiple data formats and protocols, and can flexibly adapt to the requirements of different data sources. At the same time, parallel processing and distributed storage technologies are adopted to ensure the real-time and efficient data collection. For example, when collecting multi-source data, through innovative data collection strategies, the data resources required for large model training are accurately captured and integrated from multiple sources, and various types of file resources, database table resources, and interface link resources are supported. At the same time, it also provides good support for automatic synchronization of data from Excel files to database tables with one click, which greatly simplifies the data migration process. The data collection module 11 significantly improves the efficiency and real-time performance of data collection through multi-source data integration and efficient transmission mechanism.

[0025] In some embodiments, the data collection module may include: a transmission unit, which is used to execute a breakpoint resume mechanism when collecting data resources; an encryption unit, which is communicatively connected to the transmission unit and is used to encrypt data resources during the data collection process.

[0026] During the data collection process, encryption technology is used by the encryption unit to ensure the security of data transmission. At the same time, the breakpoint resumption mechanism adopted by the transmission unit not only ensures the integrity of the data without omissions, but also significantly improves the efficiency of data upload and download, ensuring smooth data flow.

[0027] In some embodiments, data resources can be converted into data sets with higher value for subsequent data analysis and application. For example, through data collection, integration, cleaning, processing, analysis and modeling, data resources are organized in a certain format or structure to create a data set.

[0028] Continue to refer Figure 1 The data set management module 12 is responsible for unified management and maintenance of the collected data sets. The data set management module 12 can realize functions such as data set information registration / import and data resource attachment. By integrating distributed coordination services such as Zookeeper, real-time synchronization and consistency maintenance of data set information can be achieved. At the same time, message queue technologies such as Kafka are used to achieve rapid transmission and efficient processing of data set information.

[0029] In some embodiments, the data set management module may include: a registration unit, the registration unit is used to register the collected data resources as a data set; wherein the data set name is the data source, and the data set description contains detailed information of the data set; a first entry unit, the first entry unit is used to enter a single data set at a time; wherein a single data set is directly registered on the page; a second entry unit, the second entry unit is used to enter multiple data sets in words; wherein multiple data sets are filled in a standard Excel file and then uploaded and entered.

[0030] Registering a dataset means registering the created dataset in a platform or system so that other users or projects can access and use these datasets. Registering a dataset usually involves setting metadata such as the name, description, and tags of the dataset, and specifying the storage location and access rights of the dataset. By registering a dataset, dataset sharing and collaboration can be achieved, and the utilization and value of data resources can be improved. Data collection first requires entering dataset information. The dataset management module provides two entry methods to meet the needs of different scenarios: for a single dataset, you can directly register on the page through the first entry unit, and for multiple datasets, you can fill in a standard Excel file and upload it for entry through the second entry unit. When entering, you can follow the following strict strategies to ensure data quality: the dataset name is the source of the data, the dataset description should contain detailed information about the dataset, and for structured data, you also need to supplement the relevant information of the entry information items, including but not limited to the name, description, data format, data length, etc. At the same time, the dataset management module can also provide template saving to facilitate the rapid entry of similar datasets.

[0031] In some embodiments, the data set management module may include: a resource attachment unit, the resource attachment unit is used to attach data resources to data file resources or database table resources; wherein the data file resources include at least one of PDF, EXCEL, CSV and JSON, and the database table resources include at least one of MySQL, Oracle and API interface; a verification unit, the verification unit is communicatively connected to the resource attachment unit, and the verification unit is used to perform network verification and authority verification when the data is attached; a mapping unit, the mapping unit is communicatively connected to the resource attachment unit, and the mapping unit is used to perform association mapping of corresponding information items when the data is attached.

[0032] After the data set is created, you can attach resources. Currently, the available resource types include common data file resources, such as PDF, EXCEL, CSV, JSON, etc.; common database table resources, such as MySQL, Oracle, etc., and API interfaces, etc. When attaching file types, you can choose to upload files directly from the local computer. When attaching a database, you need to perform a series of verifications such as network verification and permission verification, and do the association mapping of corresponding information items.

[0033] In one embodiment, the data set management module may further include: an information change maintenance unit, which is used to synchronize changes when the data set information changes; and a resource change maintenance unit, which is used to update and supplement the data file resources or database table resources attached to the data set when the data set information changes. It can be understood that the data set information supports irregular maintenance, and timely synchronization is achieved when there are changes, and the resources attached to the data set support continuous updating and supplementation.

[0034] Continue to refer Figure 1 The data review module 13 is in communication with the data set management module 12, and is used to review the input data resources. The data review module 13 strictly reviews the data to ensure data quality and standardization.

[0035] In some embodiments, the data review module may include: a data set review unit, which is used to perform at least one of registration review, change review, offline review and release review; a process review unit, which is used to perform internal department review and company-level review; wherein the review content includes at least one of the standardization of data set registration information, whether the information items are accurate and whether the resources are complete.

[0036] First of all, data audit is divided into data set audit and data resource audit. In the two scenarios, they cover multiple links such as registration (release) audit, change audit, and offline audit to ensure the integrity and accuracy of the data. From the process point of view, data audit is divided into internal department audit and company audit. The audit content includes the standardization of data set registration information, whether the information items are accurate, whether the resources are complete, etc., to ensure the compliance and standardization of data. Therefore, based on the data set audit unit and the process audit unit, multi-dimensional audit can be achieved. For example, the input information must first pass the internal audit of the department. After passing multiple links of the internal audit of the department, it will be submitted to the company level for review. After the multiple links of the company level audit are completed, the data can be published and shared. Among them, the links of the internal audit of the department and the audit at the company level can be the same or different, depending on the actual audit needs.

[0037] For example, Figure 2 is a schematic diagram of a cross-functional data operation process provided by an exemplary embodiment of the present invention. Figure 2 As shown in the figure, in the data operation platform, the department data specialist is responsible for registering the data set, improving the data set information, uploading and attaching resources, and drafting the data catalog, and finally submitting it to the department auditor for review. The department auditor will review it. If the review is passed, it will be submitted to the platform auditor. If the review is not passed, it will be returned to the department data specialist to re-complete the data set information. The platform auditor is responsible for the platform administrator review. If the review is not passed, it will be returned to the department data specialist to re-complete the data set information. If the review is passed, it will be published and the data catalog will be generated. Then it will be transferred to the offline model engineer, who will screen the required data, obtain the data file, and finally clean, label and train the data. This completes the complete cross-functional review process of data operations.

[0038] The data audit module ensures the accuracy and integrity of data sets and information items through a multi-dimensional audit process and strict audit standards. At the same time, encryption technology and breakpoint resuming mechanism can also be used in the process of data collection, transmission and storage to ensure data security and prevent data leakage and damage.

[0039] In some embodiments, the data set review unit includes: a timeliness review subunit, which is used to review the time interval of the data set and determine whether the data set within the preset time interval is entered; an accuracy review subunit, which is used to review whether there is source information of the data set and the accuracy of the source information; a uniqueness review subunit, which is used to review the similarity between the entered data set and the existing data set. When the similarity is greater than or equal to the preset similarity, it is determined that the entered data set is duplicated and is not entered; an integrity review subunit, which is used to calculate the data missing rate of the data set. When the data missing rate is less than the preset missing rate, it is entered; a consistency review subunit, which is used to review the consistency of the data in the data set in terms of naming, format, structure and constraint rules; and a compliance review subunit, which is used to confirm that there is no sensitive data or non-compliant data in the included data through keyword retrieval technology.

[0040] Based on the multiple sub-units in the data set audit unit, specific strategies for data audit can be implemented: Audit data timeliness: According to the specific experience of the industry in which the data is located, given different time intervals, only valid data within the given time period will be entered; Audit data accuracy: The data should indicate the accurate source and be available for verification to ensure that the data content reflects the actual situation of the transaction and there is no false information; Audit data uniqueness: Repeat the data in the data set, and use advanced algorithms to perform similarity verification between the newly entered data set and the existing data set; Audit data integrity: Use big data technology to quickly calculate the data integrity rate, and data with a missing rate higher than the threshold will not be entered; Audit data consistency: The data in the same data set should be consistent in naming, format, structure and constraint rules. The data set and data file names are unified as "data source-data volume-batch", keeping the data in the data set with the same fields and structure to ensure that it can be read and processed in the same way; Audit data compliance: Use keyword retrieval technology to confirm that there is no sensitive data or non-compliant data in the included data.

[0041] Continue to refer Figure 1 The data publishing module 14 is in communication connection with the data review module 13 , and the data publishing module 14 is used to publish the data resources that have passed the data review.

[0042] In some embodiments, the data publishing module includes: a publishing unit, which is used to provide different publishing methods based on usage requirements; a data service unit, which is used to provide online data interface development and support at least one data service among data interface, data push, data direct connection and file download; a permission management and access control unit, which is communicated with the publishing unit and the data service unit, and is used to manage the usage rights and access rights of publishing and data services.

[0043] The approved data sets are published and shared through a unified interface and protocol. Different data publishing methods are provided for different user requirements, making data sharing more convenient while ensuring data security. Self-service online data interface development is provided, supporting a variety of data service methods such as data interface, data push, data direct connection and file download, so that users can obtain and use data according to their needs. At the same time, permission management and access control mechanisms are adopted to ensure data security and privacy. The data publishing module provides a variety of data publishing methods and convenient access control mechanisms through a unified interface and protocol, allowing users to quickly obtain and use data according to their needs.

[0044] In some embodiments, the system for operating training data in a large model scenario may also include: a task scheduling module, which is communicatively connected to the data collection module, the data set management module, the data review module and the data publishing module, and the task scheduling module is used to adjust the task execution frequency and task execution priority based on task requirements; a monitoring module, which is communicatively connected to the task scheduling module, and the monitoring module is used to monitor the task execution status, and when the task execution status is abnormal, an alarm notification is issued.

[0045] The task scheduling module can realize the automatic scheduling and efficient execution of data processing tasks. The task scheduling module supports flexible configuration and expansion of tasks, and can dynamically adjust the frequency and priority of task execution according to actual needs. At the same time, combined with cache technologies such as redis, the efficiency and response speed of task scheduling can be improved. In addition, the monitoring module has task monitoring and alarm functions, which can monitor the task execution status in real time and issue alarm notifications to ensure the security and stability of the data processing process.

[0046] In some embodiments, the present invention adopts modular design and plug-in expansion mechanism, so that the system has high flexibility and scalability. Users can select appropriate modules and plug-ins for combination and configuration according to their own business scenarios and data processing requirements to achieve personalized data operation solutions. At the same time, the system also supports dynamic task configuration and expansion, and can dynamically adjust the task execution frequency and priority according to actual needs, thereby improving the adaptability and flexibility of the system.

[0047] Figure 3 is a flowchart of a method for operating training data in a large model scenario provided by an exemplary embodiment of the present invention, such as Figure 3 As shown, according to another aspect of the present invention, a method for operating training data in a large model scenario is provided, which is applied to the system for operating training data in a large model scenario provided by the present invention. The method for operating training data in a large model scenario includes: Step 100: Collecting data resources from multiple source channels through a data collection module; Step 200: Registering, entering, attaching and maintaining the collected data resources through a data set management module; Step 300: Performing data audit on the entered data resources through a data review module; Step 400: Data audit includes data set audit and process audit; Step 500: Publishing data resources that have passed the data audit through a data publishing module.

[0048] The method of operating training data in large model scenarios is based on a modular system. Through automated and intelligent data processing processes, manual intervention and error rates are reduced, and the cost of data processing is reduced. At the same time, the efficiency and stability of the system also reduce the cost of maintenance and repair, further reducing the overall operating cost. Among them, through the data collection module, an efficient data collection strategy is designed to automatically collect data resources required for large model training from multiple sources, ensuring data diversity. The dataset management module can realize functions such as dataset information registration / import and data resource attachment. By integrating distributed coordination services such as Zookeeper, real-time synchronization and consistency maintenance of dataset information are achieved. At the same time, message queue technologies such as Kafka are used to achieve rapid transmission and efficient processing of dataset information. The data review module ensures the accuracy and integrity of datasets and information items through multi-dimensional review processes and strict review standards. The data publishing module provides a variety of data publishing methods and convenient access control mechanisms through unified interfaces and protocols, allowing users to quickly obtain and use data according to their needs. The method of operating training data in large model scenarios based on data collection module, data set management module, data review module and data publishing module can improve data processing efficiency, ensure data quality and security, optimize user experience and reduce operating costs. It not only provides a solid data foundation for large model training, but also promotes the automation process of data collection work, which is of great significance to improving the accuracy and efficiency of large model training.

[0049] Figure 4 is a flow chart of an operation mode of operating training data in a large model scenario provided by an exemplary embodiment of the present invention, such as Figure 4As shown, firstly, data collection is performed, and multi-source channel support is provided: database tables, data files, API interfaces and external links. Data encryption and breakpoint continuation are performed during data transmission. After data collection is completed, resources are attached, and then data information is entered: through page registration or template import. After entry is completed, it is submitted to the department data review for data information review and data resource review. After the department data review is passed, it is submitted to the company data review, and data information review and data resource review are also performed. If the department data review is rejected, data collection will be performed again. In addition, when data will be continuously updated, data information maintenance and changes will also be performed, including data information changes and data resource changes. After data information maintenance and changes are performed, it needs to be submitted to the company data review for data information review and data resource review. If the company data review is rejected, data collection will be performed again. After the company data review is passed, data release and sharing are performed. Data release and sharing include data release and data sharing.

[0050] An embodiment of the present invention provides a device for operating training data in a large model scenario. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From a hardware perspective, in addition to the CPU, memory, network interface, and non-volatile memory, the device in the embodiment where the device is located can generally include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, as a device in a logical sense, it is formed by the CPU of the device where it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution.

[0051] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program for executing the method for operating training data in a large model scenario of any of the above embodiments.

[0052] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method of operating training data in a large model scenario according to various embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0053] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: a processor; a memory for storing instructions executable by the processor; and a processor for executing the method for operating training data in a large model scenario of any of the above embodiments.

[0054] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method for operating training data in a large model scenario according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A system for operating training data in a large model scenario, characterized in that: include: A data collection module, wherein the data collection module is used to collect data resources from multiple source channels; A data set management module, the data set management module is in communication with the data collection module, and the data set management module is used to register, input, attach and maintain the collected data resources; A data review module, which is in communication with the data set management module and is used to review the input data resources; the data review includes data set review and process review; A data publishing module is communicatively connected with the data review module, and is used for publishing data resources that have passed the data review.

2. The system for operating training data in a large model scenario according to claim 1, characterized in that: The data collection module comprises: A transmission unit, the transmission unit is used to execute a breakpoint resume mechanism when collecting data resources; An encryption unit is communicatively connected to the transmission unit, and is used to encrypt data resources during the data collection process.

3. The system for operating training data in a large model scenario according to claim 1, characterized in that: The data set management module includes: A registration unit, the registration unit is used to register the collected data resources as a data set; wherein the data set name is the data source, and the data set description includes detailed information of the data set; A first input unit, the first input unit is used to input a single data set at a time; wherein the single data set is directly registered on the page; The second input unit is used to input multiple data sets by word; wherein the multiple data sets are filled in a standard Excel file and then uploaded and input.

4. The system for operating training data in a large model scenario according to claim 1, characterized in that: The data set management module includes: A resource attachment unit, the resource attachment unit is used to attach a data resource to a data file resource or a database table resource; wherein the data file resource includes at least one of PDF, EXCEL, CSV and JSON, and the database table resource includes at least one of MySQL, Oracle and an API interface; A verification unit, the verification unit is in communication with the resource attachment unit, and the verification unit is used to perform network authentication and authority authentication when data is attached; A mapping unit is communicatively connected with the resource mounting unit, and is used for making association mapping of corresponding information items when data is mounted.

5. The system for operating training data in a large model scenario according to claim 4, characterized in that: The data set management module also includes: An information change maintenance unit, wherein the information change maintenance unit is used to synchronize changes when the data set information is changed; The resource change maintenance unit is used to update and supplement the data file resources or database table resources attached to the data set when the data set information changes.

6. The system for operating training data in a large model scenario according to claim 1, characterized in that: The data review module includes: A data set review unit, the data set review unit is used to perform at least one of a registration review, a change review, a offline review, and a release review; A process-based audit unit, which is used to conduct internal department audits and company-level audits; The audit content includes at least one of the following: the standardization of the dataset registration information, the accuracy of the information items, and the completeness of the resources.

7. The system for operating training data in a large model scenario according to claim 6, characterized in that: The data set review unit includes: A timeliness review subunit, which is used to review the time interval of the data set and determine that the data set within the preset time interval is entered; An accuracy review subunit, the accuracy review subunit is used to review whether the data set has source information and the accuracy of the source information; A uniqueness review subunit, which is used to review the similarity between the input data set and the existing data set. When the similarity is greater than or equal to a preset similarity, the input data set is determined to be duplicated and is not input; An integrity audit subunit, which is used to calculate the data missing rate of the data set, and enter the data when the data missing rate is less than a preset missing rate; A consistency review subunit, which is used to review the consistency of the data in the data set in terms of naming, format, structure and constraint rules; The compliance review sub-unit is used to confirm that there is no sensitive data or non-compliant data in the collected data through keyword retrieval technology.

8. The system for operating training data in a large model scenario according to claim 1, characterized in that: The data publishing module includes: A publishing unit, the publishing unit is used to provide different publishing modes based on usage requirements; A data service unit, the data service unit is used to provide online data interface development, and supports at least one data service of data interface, data push, data direct connection and file download; A rights management and access control unit is communicatively connected with the publishing unit and the data service unit, and is used to manage the use rights and access rights of publishing and data services.

9. The system for operating training data in a large model scenario according to claim 1, characterized in that: Systems that operate training data in large model scenarios also include: A task scheduling module, wherein the task scheduling module is in communication with the data collection module, the data set management module, the data review module and the data publishing module, and the task scheduling module is used to adjust the task execution frequency and task execution priority based on task requirements; A monitoring module is connected to the task scheduling module for communication, and is used to monitor the task execution status and issue an alarm notification when the task execution status is abnormal.

10. A method for operating training data in a large model scenario, applied to a system for operating training data in a large model scenario as claimed in any one of claims 1 to 9, characterized in that: Methods for operating training data in large model scenarios include: Collect data resources from multiple sources through the data collection module; Through the data set management module, the collected data resources are registered, entered, linked and maintained; Through the data review module, the entered data resources are reviewed; the data review includes data set review and process review; Data resources that have passed data review will be published through the data publishing module.

Images