Data management method and system based on large model
By automating the data acquisition, cleaning, quality inspection, and uploading processes, the problem of low data quality in traditional data management has been solved, and efficient data management and adaptation of large model training data have been achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-22
- Publication Date
- 2026-03-31
AI Technical Summary
Traditional data management methods rely on manual processes, resulting in low data quality that is susceptible to subjective judgment biases and operational errors, making it difficult to meet the high-quality data requirements for training large industry models.
By employing a large-model-based data management approach, including data acquisition, cleaning, quality inspection, and uploading processes, data is processed automatically to generate datasets with associated version numbers, ensuring data quality and historical traceability.
It achieves intelligent data management, significantly improves data quality, requires no manual intervention, and is suitable for large model training needs.
Smart Images
Figure CN121764907A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a data management method and system based on large models. Background Technology
[0002] With the rapid development of digitalization and artificial intelligence technologies, data has become a crucial production factor in the digital economy era. During the intelligent transformation of industries, large-scale models, through deep learning and big data analytics, can uncover deep-seated patterns and potential value within data, providing decision support and business optimization solutions for the industry.
[0003] Currently, traditional data management still relies on manual methods, which require manual screening of raw data that meets the requirements, manual cleaning, and manual recording of data flow information.
[0004] However, training large industry models requires high-quality data as support. Traditional methods have certain limitations. Staff are easily affected by subjective judgment bias and operational errors during data screening, cleaning and quality inspection, resulting in low data quality. Summary of the Invention
[0005] This invention provides a data management method and system based on large models, which can improve data quality.
[0006] In a first aspect, embodiments of the present invention provide a data management method based on a large model, the method comprising:
[0007] Based on the data acquisition request submitted by the current user, collect the raw data that matches the data acquisition request and register it as the raw dataset associated with the original version number;
[0008] Obtain the current cleaning request submitted by the current user, perform at least one cleaning operation on the original dataset to be cleaned using a preset cleaning strategy, and register it as a cleaned dataset with an associated cleaning version number.
[0009] Perform at least one quality inspection operation on the cleaned dataset, and register the cleaned dataset as a sample dataset with the associated sample version number after it meets the preset quality inspection standards;
[0010] An uploading operation is performed on the sample dataset to train the large model using the uploaded sample dataset.
[0011] Preferably,
[0012] The process of collecting and registering raw data that matches the current user's data acquisition request as an original dataset associated with the original version number includes:
[0013] Obtain the data acquisition request submitted by the current user, wherein the data acquisition request includes at least: request title, request document, responsible person, and planned start and end dates;
[0014] Based on the data acquisition requirements, appropriate raw data is collected, and the collected raw data is preprocessed to generate an original dataset.
[0015] The dataset attributes of the original dataset are set to correspond to the original version number, and the original dataset and the corresponding original version number are associated and synchronously registered to the data platform.
[0016] Preferably,
[0017] The process of obtaining the current cleaning request submitted by the current user, performing at least one cleaning operation on the original dataset to be cleaned using a preset cleaning strategy, and registering it as a cleaned dataset with an associated cleaning version number includes:
[0018] Obtain the current cleaning request submitted by the current user, and configure a cleaning priority for the current cleaning request based on the preset cleaning strategy;
[0019] Based on the cleaning priority, the preset cleaning strategy is used to perform at least one cleaning operation on the original dataset to be cleaned, wherein the cleaning operation includes: data deduplication, format unification and error value correction;
[0020] The cleaning progress of the original dataset to be cleaned is monitored in real time and the cleaning status is marked, wherein the cleaning status marks include: not cleaned, cleaning in progress, and cleaned;
[0021] Extract the datasets marked as cleaned, set the corresponding cleaning version number, associate the cleaned datasets with the corresponding cleaning version number, and synchronously register them to the data platform.
[0022] Preferably,
[0023] The step of performing at least one quality inspection operation on the cleaned dataset and registering it as a sample dataset with an associated sample version number after the cleaned dataset meets the preset quality inspection standards includes:
[0024] D1: Based on the preset quality inspection standard, perform at least one quality inspection operation on the cleaned dataset and mark the quality inspection status, wherein the quality inspection status marks include: not inspected, in process of quality inspection, and inspected.
[0025] D2: Filter out the datasets marked as "inspected" and determine whether the inspected datasets meet the preset quality inspection standards. If yes, proceed to step D4; otherwise, proceed to step D3.
[0026] D3: Update the cleaned dataset based on the quality inspection rejection factors, and return to the execution step D1;
[0027] D4: Based on the name of the cleaned dataset, the quality inspection status flag, and the creation time, set the corresponding sample version number for the quality inspected dataset;
[0028] D5: Associate the quality-inspected dataset with the corresponding sample version number, and register it as the sample dataset and store it in the data platform.
[0029] Preferably,
[0030] The step of performing an upload operation on the sample dataset to train a large model using the uploaded sample dataset includes:
[0031] Obtain the sample dataset to be uploaded and perform a registration confirmation operation on the sample dataset to be uploaded, wherein the registration confirmation includes: the sample version number associated with the sample dataset, the data query method, the data type and the dataset size;
[0032] When the registration is confirmed as successful, the uploading operation is performed on the sample dataset to be uploaded and the sample dataset to be uploaded is stored in the large model training storage area;
[0033] The status of the sample dataset is updated synchronously to "listed" and the listing operation information corresponding to the sample dataset is recorded. The listing operation information includes: the person who listed the dataset, the listing time, the storage path, and the sample version number.
[0034] Secondly, embodiments of the present invention provide a data management system based on a large model, the system comprising:
[0035] Data acquisition module: used to collect raw data that matches the data acquisition request submitted by the current user and register it as a raw dataset associated with the original version number;
[0036] Data cleaning module: used to obtain the current cleaning request submitted by the current user, perform at least one cleaning operation on the original dataset to be cleaned obtained by the data acquisition module using a preset cleaning strategy, and register it as a cleaned dataset with an associated cleaning version number;
[0037] Data quality inspection module: used to perform at least one quality inspection operation on the cleaned dataset generated by the data cleaning module, and register the cleaned dataset as a sample dataset with an associated sample version number after it meets the preset quality inspection standards;
[0038] Data uploading module: Used to perform uploading operations on the sample dataset generated by the data quality inspection module, so as to train the large model using the uploaded sample dataset.
[0039] Preferably,
[0040] The data acquisition module is used to perform:
[0041] Obtain the data acquisition request submitted by the current user, wherein the data acquisition request includes at least: request title, request document, responsible person, and planned start and end dates;
[0042] Based on the data acquisition requirements, appropriate raw data is collected, and the collected raw data is preprocessed to generate an original dataset.
[0043] The dataset attributes of the original dataset are set to correspond to the original version number, and the original dataset and the corresponding original version number are associated and synchronously registered to the data platform.
[0044] Preferably,
[0045] The data cleaning module is used to perform:
[0046] Obtain the current cleaning request submitted by the current user, and configure a cleaning priority for the current cleaning request based on the preset cleaning strategy;
[0047] Based on the cleaning priority, the preset cleaning strategy is used to perform at least one cleaning operation on the original dataset to be cleaned, wherein the cleaning operation includes: data deduplication, format unification and error value correction;
[0048] The cleaning progress of the original dataset to be cleaned is monitored in real time and the cleaning status is marked, wherein the cleaning status marks include: not cleaned, cleaning in progress, and cleaned;
[0049] Extract the datasets marked as cleaned, set the corresponding cleaning version number, associate the cleaned datasets with the corresponding cleaning version number, and synchronously register them to the data platform.
[0050] Preferably,
[0051] The data quality inspection module is used to perform:
[0052] D1: Based on the preset quality inspection standard, perform at least one quality inspection operation on the cleaned dataset and mark the quality inspection status, wherein the quality inspection status marks include: not inspected, in process of quality inspection, and inspected.
[0053] D2: Filter out the datasets marked as "inspected" and determine whether the inspected datasets meet the preset quality inspection standards. If yes, proceed to step D4; otherwise, proceed to step D3.
[0054] D3: Update the cleaned dataset based on the quality inspection rejection factors, and return to the execution step D1;
[0055] D4: Based on the name of the cleaned dataset, the quality inspection status flag, and the creation time, set the corresponding sample version number for the quality inspected dataset;
[0056] D5: Associate the quality-inspected dataset with the corresponding sample version number, and register it as the sample dataset and store it in the data platform.
[0057] Preferably,
[0058] The data uploading module is used to perform:
[0059] Obtain the sample dataset to be uploaded and perform a registration confirmation operation on the sample dataset to be uploaded, wherein the registration confirmation includes: the sample version number associated with the sample dataset, the data query method, the data type and the dataset size;
[0060] When the registration is confirmed as successful, the uploading operation is performed on the sample dataset to be uploaded and the sample dataset to be uploaded is stored in the large model training storage area;
[0061] The status of the sample dataset is updated synchronously to "listed" and the listing operation information corresponding to the sample dataset is recorded. The listing operation information includes: the person who listed the dataset, the listing time, the storage path, and the sample version number.
[0062] This invention provides a data management method and system based on a large model. The method first collects suitable raw data based on the current user's data acquisition needs and registers the corresponding raw version number, laying the data foundation for subsequent data processing. Next, when the current user submits a data cleaning request, multiple iterative cleaning operations are performed on the raw dataset to be cleaned to optimize data quality, while the corresponding cleaning version number is registered to ensure the traceability of data history and rollback requirements. Then, multiple quality inspection operations are performed on the cleaned dataset to ensure data quality, and it is registered as a sample dataset with an associated sample version number for subsequent retrieval operations. Finally, the sample dataset is uploaded to the system to complete subsequent data annotation and training of the large model. This process eliminates the need for manual intervention, achieving intelligent data management and significantly improving data quality. Attached Figure Description
[0063] Figure 1 This is a flowchart of a data management method based on a large model provided in an embodiment of the present invention;
[0064] Figure 2 This is a flowchart of another data management method based on a large model provided in an embodiment of the present invention;
[0065] Figure 3 This is a schematic diagram of a data management system based on a large model provided in an embodiment of the present invention. Detailed Implementation
[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0067] like Figure 1 As shown, this embodiment of the invention provides a data management method based on a large model, which may include the following steps:
[0068] Step 101: Based on the data acquisition request submitted by the current user, collect the raw data that matches the data acquisition request and register it as the raw dataset associated with the original version number;
[0069] Step 102: Obtain the current cleaning request submitted by the current user, perform at least one cleaning operation on the original dataset to be cleaned using the preset cleaning strategy, and register it as a cleaned dataset with the associated cleaning version number;
[0070] Step 103: Perform at least one quality inspection operation on the cleaned dataset, and register the cleaned dataset as a sample dataset with the associated sample version number after it meets the preset quality inspection standards;
[0071] Step 104: Perform an upload operation on the sample dataset to train the large model using the uploaded sample dataset.
[0072] This invention provides a data management method based on a large model. The method first collects suitable raw data based on the current user's data acquisition needs and registers the corresponding raw version number, laying the data foundation for subsequent data processing. Next, when the current user submits a data cleaning request, multiple iterative cleaning operations are performed on the raw dataset to be cleaned to optimize data quality, while the corresponding cleaning version number is registered to ensure the traceability of data history and rollback requirements. Then, multiple quality inspection operations are performed on the cleaned dataset to ensure data quality, and it is registered as a sample dataset with an associated sample version number for subsequent retrieval operations. Finally, the sample dataset is uploaded to the platform to complete subsequent data annotation and training of the large model. This process eliminates the need for manual intervention, achieving intelligent data management and significantly improving data quality.
[0073] In order to obtain the original data required for the adaptation data acquisition, in one embodiment of the present invention, step 101 in the above embodiment may specifically include the following steps:
[0074] Obtain the data acquisition request submitted by the current user, wherein the data acquisition request includes at least: request title, request document, responsible person, and planned start and end dates;
[0075] Based on the data acquisition requirements, appropriate raw data is collected, and the collected raw data is preprocessed to generate an original dataset.
[0076] The dataset attributes of the original dataset are set to correspond to the original version number, and the original dataset and the corresponding original version number are associated and synchronously registered to the data platform.
[0077] In this embodiment of the invention, in order to obtain the original data that matches the data acquisition requirements, when the data acquisition requirements submitted by the current user are obtained, the original data that matches them can be collected. Since the collected original data have different formats, they can be preprocessed to generate the original dataset. Then, based on the dataset attributes of the original dataset, the corresponding original version number is set so that data can be retrieved through the original version number, thereby ensuring the traceability of data history and the rollback requirements.
[0078] To optimize data quality, in one embodiment of the present invention, step 102 in the above embodiment may specifically include the following steps:
[0079] Obtain the current cleaning request submitted by the current user, and configure a cleaning priority for the current cleaning request based on the preset cleaning strategy;
[0080] Based on the cleaning priority, the preset cleaning strategy is used to perform at least one cleaning operation on the original dataset to be cleaned, wherein the cleaning operation includes: data deduplication, format unification and error value correction;
[0081] The cleaning progress of the original dataset to be cleaned is monitored in real time and the cleaning status is marked, wherein the cleaning status marks include: not cleaned, cleaning in progress, and cleaned;
[0082] Extract the datasets marked as cleaned, set the corresponding cleaning version number, associate the cleaned datasets with the corresponding cleaning version number, and synchronously register them to the data platform.
[0083] In this embodiment of the invention, in order to optimize data quality, the original dataset needs to be cleaned. After obtaining the current user's current cleaning needs, the cleaning priority can be configured based on a preset cleaning strategy (e.g., business needs and data importance) to ensure that urgent needs are handled first. Then, based on the set cleaning priority, multiple cleaning operations (e.g., data deduplication, format unification, and error value correction) are performed on the original dataset to be cleaned to continuously optimize data quality. Next, the cleaning progress is marked with a cleaning status (e.g., not cleaned, cleaning in progress, and cleaned) to facilitate the current user's identification of the cleaning status and data management. Finally, the cleaned dataset is extracted and a corresponding cleaning version number is set to achieve data history traceability.
[0084] To further improve data quality, in one embodiment of the present invention, step 103 in the above embodiment may specifically include the following steps:
[0085] D1: Based on the preset quality inspection standard, perform at least one quality inspection operation on the cleaned dataset and mark the quality inspection status, wherein the quality inspection status marks include: not inspected, in process of quality inspection, and inspected.
[0086] D2: Filter out the datasets marked as "inspected" and determine whether the inspected datasets meet the preset quality inspection standards. If yes, proceed to step D4; otherwise, proceed to step D3.
[0087] D3: Update the cleaned dataset based on the quality inspection rejection factors, and return to the execution step D1;
[0088] D4: Based on the name of the cleaned dataset, the quality inspection status flag, and the creation time, set the corresponding sample version number for the quality inspected dataset;
[0089] D5: Associate the quality-inspected dataset with the corresponding sample version number, and register it as the sample dataset and store it in the data platform.
[0090] In this embodiment of the invention, to further improve data quality, the cleaned dataset needs to undergo quality inspection. First, based on preset quality inspection standards, multiple quality inspection operations are performed on the cleaned dataset, and the quality inspection status is marked (e.g., not inspected, under inspection, inspected). This allows current users to quickly identify the data quality inspection status and manage it effectively. Then, datasets marked as "inspected" are selected, and corresponding sample version numbers are set when they meet the preset quality inspection standards to track historical data changes and meet potential rollback requirements. When they do not meet the preset quality inspection standards, the rejection reason is obtained, and the cleaned dataset is updated for re-inspection. Finally, the inspected datasets are registered as sample datasets and stored on the data platform. This ensures that the inspected data is properly managed and effectively utilized, thereby effectively improving data management efficiency.
[0091] To improve the quality of large model training data, in one embodiment of the present invention, step 104 in the above embodiment may specifically include the following steps:
[0092] Obtain the sample dataset to be uploaded and perform a registration confirmation operation on the sample dataset to be uploaded, wherein the registration confirmation includes: the sample version number associated with the sample dataset, the data query method, the data type and the dataset size;
[0093] When the registration is confirmed as successful, the uploading operation is performed on the sample dataset to be uploaded and the sample dataset to be uploaded is stored in the large model training storage area;
[0094] The status of the sample dataset is updated synchronously to "listed" and the listing operation information corresponding to the sample dataset is recorded. The listing operation information includes: the person who listed the dataset, the listing time, the storage path, and the sample version number.
[0095] In this embodiment of the invention, in order to improve the quality of large model training data, the sample dataset to be uploaded can be registered and confirmed first (e.g., the sample version number associated with the sample dataset, data query method, data type, and dataset size) to improve data accuracy through repeated verification of the sample dataset. Then, when the registration and confirmation are passed, the sample dataset to be uploaded is uploaded and stored in the large model training storage area for subsequent data annotation, large model training, and other tasks. At the same time, the status can be updated to "uploaded" and the uploading operation information (e.g., uploader, uploading time, storage path, and sample version number) can be recorded to achieve subsequent data tracking and management.
[0096] like Figure 2 As shown, in order to more clearly illustrate the technical solution and advantages of the present invention, the following provides a detailed description of the data management method based on a large model, which may include the following steps:
[0097] Step 201: Based on the data acquisition request submitted by the current user, collect the raw data that matches the data acquisition request and register it as the raw dataset associated with the original version number; Step 202: Obtain the data acquisition request submitted by the current user, wherein the data acquisition request includes at least: request title, request document, responsible person, and planned start and end dates;
[0098] Step 203: Collect suitable raw data based on data acquisition requirements, and perform data preprocessing on the collected raw data to generate the raw dataset;
[0099] Specifically, the original dataset should include at least: dataset name, dataset type, storage path, dataset size, person in charge, creation time, and update time;
[0100] First, the dataset name must accurately reflect the dataset content and ensure uniqueness; second, select the corresponding dataset type (e.g., raw dataset, sample dataset); third, ensure the data's storage location on the data platform has a clear and traceable path; fourth, datasets of varying sizes should support editing and display by the current user to adapt to different storage scenarios; fifth, clearly record the creator and responsible person of the data, assigning responsibility to specific individuals; and finally, record the data's creation and update times to facilitate tracking data changes.
[0101] Step 204: Set the corresponding original version number based on the dataset attributes of the original dataset, and associate and synchronize the original dataset and the corresponding original version number to the data platform;
[0102] Step 205: Obtain the current cleaning request submitted by the current user, and configure the cleaning priority for the current cleaning request based on the preset cleaning strategy;
[0103] Step 206: Based on the cleaning priority, perform at least one cleaning operation on the original dataset to be cleaned using a preset cleaning strategy. The cleaning operation includes: data deduplication, format unification, and error value correction.
[0104] Step 207: Monitor the cleaning progress of the original dataset to be cleaned in real time and mark the cleaning status. The cleaning status marks include: not cleaned, cleaning in progress, and cleaned.
[0105] Step 208: Extract the datasets marked as cleaned, set the corresponding cleaning version number, associate the cleaned datasets with the corresponding cleaning version number, and synchronously register them to the data platform;
[0106] Step 209: Based on the preset quality inspection standards, perform at least one quality inspection operation on the cleaned dataset and mark the quality inspection status. The quality inspection status marks include: not inspected, in process of quality inspection, and inspected.
[0107] Step 210: Filter out the datasets marked as "inspected" and determine whether the inspected datasets meet the preset quality inspection standards. If yes, proceed to step 212; otherwise, proceed to step 211.
[0108] Step 211: Update the cleaned dataset based on the quality inspection rejection factors, and return to execute step 209;
[0109] For example, if a quality inspection rejection is due to failure to meet the preset quality inspection standards, and the quality inspection operation is performed again, it means that the quality inspector has performed the quality inspection operation multiple times. The quality inspection rejection is edited and corrected multiple times to ensure the accuracy of the quality inspection results.
[0110] Specifically, once a quality inspection task has been submitted, the reason for rejection will be locked, and any subsequent modification will be prohibited. This is to ensure the immutability and traceability of the quality inspection results and prevent data from being arbitrarily tampered with.
[0111] Step 212: Based on the name, quality inspection status flag, and creation time of the cleaned dataset, set the corresponding sample version number for the quality inspected dataset;
[0112] Step 213: Associate the quality-inspected dataset with the corresponding sample version number, and register it as a sample dataset for storage on the data platform;
[0113] Step 214: Obtain the sample dataset to be uploaded and perform a registration confirmation operation on the sample dataset to be uploaded. The registration confirmation includes: the sample version number associated with the sample dataset, the data query method, the data type, and the dataset size.
[0114] Specifically, the registration and confirmation process includes: filling in the dataset name and clearly identifying the dataset content; selecting the data type (e.g., supervised data, unsupervised data); and selecting the appropriate data file type based on the data file type (e.g., table, Parquet, JSON, JSONL).
[0115] For example, the data query method can be set as follows: when you enter the corresponding field (e.g., original version number, cleaned version number, sample version number), the data platform will automatically filter out the data for subsequent data processing and analysis.
[0116] Step 215: When the registration is confirmed as passed, perform the uploading operation on the sample dataset to be uploaded and store the sample dataset to be uploaded to the large model training storage area;
[0117] Step 216: Synchronously update the label status of the sample dataset to "listed" and record the listing operation information corresponding to the sample dataset. The listing operation information includes: the person who listed the dataset, the listing time, the storage path, and the sample version number.
[0118] like Figure 3 As shown, this embodiment of the invention provides a data management system based on a large model, the system comprising:
[0119] Data acquisition module 301: Used to collect raw data that matches the data acquisition request submitted by the current user and register it as a raw dataset associated with the original version number;
[0120] Data cleaning module 302: Obtains the current cleaning request submitted by the current user, performs at least one cleaning operation on the original dataset to be cleaned obtained by the data acquisition module using a preset cleaning strategy, and registers it as a cleaned dataset with an associated cleaning version number;
[0121] Data quality inspection module 303: performs at least one quality inspection operation on the cleaned dataset, and registers the cleaned dataset generated by the data cleaning module as a sample dataset with an associated sample version number after it meets the preset quality inspection standards;
[0122] Data uploading module 304: Performs uploading operation on the sample dataset generated by the data quality inspection module, so as to train the large model using the uploaded sample dataset.
[0123] like Figure 3 As shown, the data acquisition module 301 is used to perform:
[0124] Obtain the data acquisition request submitted by the current user, wherein the data acquisition request includes at least: request title, request document, responsible person, and planned start and end dates;
[0125] Based on the data acquisition requirements, appropriate raw data is collected, and the collected raw data is preprocessed to generate an original dataset.
[0126] The dataset attributes of the original dataset are set to correspond to the original version number, and the original dataset and the corresponding original version number are associated and synchronously registered to the data platform.
[0127] like Figure 3 As shown, the data cleaning module 302 is used to perform:
[0128] Obtain the current cleaning request submitted by the current user, and configure a cleaning priority for the current cleaning request based on the preset cleaning strategy;
[0129] Based on the cleaning priority, the preset cleaning strategy is used to perform at least one cleaning operation on the original dataset to be cleaned, wherein the cleaning operation includes: data deduplication, format unification and error value correction;
[0130] The cleaning progress of the original dataset to be cleaned is monitored in real time and the cleaning status is marked, wherein the cleaning status marks include: not cleaned, cleaning in progress, and cleaned;
[0131] Extract the datasets marked as cleaned, set the corresponding cleaning version number, associate the cleaned datasets with the corresponding cleaning version number, and synchronously register them to the data platform.
[0132] like Figure 3 As shown, the data quality inspection module 303 is used to perform:
[0133] D1: Based on the preset quality inspection standard, perform at least one quality inspection operation on the cleaned dataset and mark the quality inspection status, wherein the quality inspection status marks include: not inspected, in process of quality inspection, and inspected.
[0134] D2: Filter out the datasets marked as "inspected" and determine whether the inspected datasets meet the preset quality inspection standards. If yes, proceed to step D4; otherwise, proceed to step D3.
[0135] D3: Update the cleaned dataset based on the quality inspection rejection factors, and return to the execution step D1;
[0136] D4: Based on the name of the cleaned dataset, the quality inspection status flag, and the creation time, set the corresponding sample version number for the quality inspected dataset;
[0137] D5: Associate the quality-inspected dataset with the corresponding sample version number, and register it as the sample dataset and store it in the data platform.
[0138] like Figure 3 As shown, the data uploading module 304 is used to perform:
[0139] Obtain the sample dataset to be uploaded and perform a registration confirmation operation on the sample dataset to be uploaded, wherein the registration confirmation includes: the sample version number associated with the sample dataset, the data query method, the data type and the dataset size;
[0140] When the registration is confirmed as successful, the uploading operation is performed on the sample dataset to be uploaded and the sample dataset to be uploaded is stored in the large model training storage area;
[0141] The status of the sample dataset is updated synchronously to "listed" and the listing operation information corresponding to the sample dataset is recorded. The listing operation information includes: the person who listed the dataset, the listing time, the storage path, and the sample version number.
[0142] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the large-model-based data management system. In other embodiments of the present invention, the large-model-based data management system may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0143] The information interaction and execution process between the various units in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.
[0144] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.
[0145] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.
[0146] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.
[0147] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0148] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0149] The various embodiments of the present invention have at least the following beneficial effects:
[0150] 1. This invention provides a data management method based on a large model. The method first collects suitable raw data based on the current user's data acquisition needs and registers the corresponding raw version number, laying the data foundation for subsequent data processing. Next, when the current user submits a data cleaning request, multiple iterative cleaning operations are performed on the raw dataset to be cleaned to optimize data quality, while the corresponding cleaning version number is registered to ensure the traceability of data history and rollback requirements. Then, multiple quality inspection operations are performed on the cleaned dataset to ensure data quality, and it is registered as a sample dataset with an associated sample version number for subsequent retrieval operations. Finally, an uploading operation is performed on the sample dataset to utilize the uploaded sample dataset for subsequent data annotation and training of the large model. The above process does not rely on manual intervention, achieving intelligent data management and significantly improving data quality.
[0151] 2. In this embodiment of the invention, in order to obtain the original data that matches the data acquisition requirements, when the data acquisition requirements submitted by the current user are obtained, the original data that matches them can be collected. Since the collected original data have different formats, they can be preprocessed to generate the original dataset. Then, based on the dataset attributes of the original dataset, the corresponding original version number is set so that data can be retrieved through the original version number, thereby ensuring the traceability of data history and the rollback requirements.
[0152] 3. In this embodiment of the invention, in order to optimize data quality, the original dataset needs to be cleaned. After obtaining the current user's current cleaning needs, the cleaning priority can be configured based on the preset cleaning strategy to ensure that urgent needs are handled first. Then, based on the set cleaning priority, multiple cleaning operations are performed on the original dataset to be cleaned, aiming to continuously optimize data quality. Next, the cleaning progress is marked with a cleaning status so that the current user can identify the cleaning status and manage the data. Finally, the cleaned dataset is extracted and a corresponding cleaning version number is set to achieve the traceability of data history.
[0153] It should be noted that not all steps and modules in the above processes and system structure diagrams are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structure described in the above embodiments can be a physical structure or a logical structure. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.
[0154] In the above embodiments, the hardware units can be implemented mechanically or electrically. For example, a hardware unit may include permanent dedicated circuitry or logic (such as a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. The hardware unit may also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor), which can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical, dedicated permanent circuitry, or temporarily configured circuitry) can be determined based on cost and time considerations.
[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A large model-based data management method, characterized by, The method comprises: Based on the current user submitted data acquisition requirements, collect and register the raw data set associated with the raw version number; Obtain the current cleaning requirements submitted by the current user, perform at least one cleaning operation on the raw data set to be cleaned using the preset cleaning strategy, and register the cleaned data set associated with the cleaning version number; Perform at least one quality inspection operation on the cleaned data set, and register the sample data set associated with the sample version number after the cleaned data set meets the preset quality inspection standard; Perform the on-shelf operation on the sample data set to train the large model using the on-shelf sample data set.
2. The method of claim 1, wherein: Based on the current user submitted data acquisition requirements, collect and register the raw data set associated with the raw version number, comprising: Obtain the data acquisition requirements submitted by the current user, wherein the data acquisition requirements at least include: requirement title, requirement file, responsible person, planned start and end date; Collect and preprocess the collected raw data based on the data acquisition requirements, and generate a raw data set; Set the corresponding raw version number based on the data set attribute of the raw data set, and synchronously register the raw data set and the corresponding raw version number to the data platform.
3. The method of claim 1, wherein: Obtain the current cleaning requirements submitted by the current user, perform at least one cleaning operation on the raw data set to be cleaned using the preset cleaning strategy, and register the cleaned data set associated with the cleaning version number, comprising: Obtain the current cleaning requirements submitted by the current user, and configure the cleaning priority of the current cleaning requirements based on the preset cleaning strategy; Based on the cleaning priority, perform the at least one cleaning operation on the raw data set to be cleaned using the preset cleaning strategy, wherein the cleaning operation includes: data deduplication, format unification and error value correction; Real-time monitoring of the cleaning progress of the raw data set to be cleaned and cleaning state marking, wherein the cleaning state marking includes: not cleaned, cleaning and cleaned; Set the corresponding cleaning version number for the data set with the cleaning state marking as cleaned, and synchronously register the cleaned data set and the corresponding cleaning version number to the data platform.
4. The method of claim 1, wherein: Perform at least one quality inspection operation on the cleaned data set, and register the sample data set associated with the sample version number after the cleaned data set meets the preset quality inspection standard, comprising: D1: Based on the preset quality inspection standard, perform at least one quality inspection operation on the cleaned data set and mark the quality inspection state, wherein the quality inspection state marking includes: not inspected, in inspection, and inspected. D2: filter out the data set with the quality inspection state marker as having been quality inspected, and judge whether the data set having been quality inspected meets the preset quality inspection standard based on the preset quality inspection standard, if yes, execute step D4, otherwise, execute step D3; D3: update the cleaned data set based on the quality inspection rejection reason, and return to execute step D1; D4: set the sample version number corresponding to the data set having been quality inspected based on the name, the quality inspection state marker and the creation time of the cleaned data set; D5: associate the data set having been quality inspected with the sample version number corresponding thereto, and register the sample data set to be stored to the data platform.
5. The method of any one of claims 1-4, wherein the performing the shelving operation on the sample data set to train the large model using the shelved sample data set comprises: obtaining a sample data set to be shelved, and performing a registration confirmation operation on the sample data set to be shelved, wherein the registration confirmation comprises the sample version number associated with the sample data set, a data query method, a data type, and a data set size; when the registration confirmation is passed, performing a shelving operation on the sample data set to be shelved and storing the sample data set to be shelved to a large model training storage area; and synchronously updating a marker state of the sample data set to shelved, and recording shelving operation information corresponding to the sample data set, wherein the shelving operation information comprises a shelving person, a shelving time, a storage path, and a sample version number. The system comprises: a data acquisition module configured to collect and register raw data associated with a raw version number based on a data acquisition requirement submitted by a current user; a data cleaning module configured to obtain a current cleaning requirement submitted by the current user, perform at least one cleaning operation on the raw data set to be cleaned obtained by the data acquisition module using a preset cleaning strategy, and register the cleaned data set associated with a cleaning version number; a data quality inspection module configured to perform at least one quality inspection operation on the cleaned data set generated by the data cleaning module, and register the sample data set associated with a sample version number after the cleaned data set meets a preset quality inspection standard; 6. A large model-based data management system, characterized by, a data shelving module configured to perform a shelving operation on the sample data set generated by the data quality inspection module to train a large model using the shelved sample data set.
7. The system of claim 6, wherein the data acquisition module is configured to: obtain the data acquisition requirement submitted by the current user, wherein the data acquisition requirement at least comprises a requirement title, a requirement file, a person in charge, a planned start and end date; collect and generate a raw data set based on the raw data collected from the data acquisition requirement and data preprocessing of the raw data. set the raw version number corresponding to the raw data set based on a data set attribute of the raw data set, and synchronously register the raw data set and the raw version number corresponding thereto to the data platform.
8. The system of claim 6, wherein The data cleaning module is configured to perform: obtaining the current cleaning requirement submitted by the current user, and configuring a cleaning priority for the current cleaning requirement based on the preset cleaning strategy; performing the at least one cleaning operation on the to-be-cleaned original data set based on the cleaning priority and the preset cleaning strategy, wherein the cleaning operation includes data deduplication, format unification, and error value correction; monitoring the cleaning progress of the to-be-cleaned original data set in real time and marking the cleaning state, wherein the cleaning state marking includes: not cleaned, cleaning, and cleaned; setting a corresponding cleaning version number for the data set with the cleaning state marking as cleaned, associating the cleaned data set and the corresponding cleaning version number, and synchronously registering them to the data platform.
9. The system of claim 6, wherein the data quality inspection module is configured to perform: D1: performing at least one quality inspection operation on the cleaned data set based on the preset quality inspection standard and marking the quality inspection state, wherein the quality inspection state marking includes: not inspected, in inspection, and inspected; D2: screening out the data set with the quality inspection state marking as inspected, and judging whether the inspected data set meets the preset quality inspection standard based on the preset quality inspection standard, if yes, performing step D4, otherwise, performing step D3; D3: updating the cleaned data set based on the quality inspection rejection reason, and returning to perform step D1; D4: setting a corresponding sample version number for the inspected data set based on the name of the cleaned data set, the quality inspection state marking, and the creation time; D5: associating the inspected data set and the corresponding sample version number, and registering the sample data set to the data platform.
10. The system of any one of claims 6-9, wherein the data shelving module is configured to perform: obtaining a to-be-shelved sample data set, and performing a registration confirmation operation on the to-be-shelved sample data set, wherein the registration confirmation includes the sample version number associated with the sample data set, a data query method, a data type, and a data set size; when the registration confirmation is passed, performing a shelving operation on the to-be-shelved sample data set and storing the to-be-shelved sample data set to a large model training storage area; synchronously updating the marking state of the sample data set to shelved, and recording the shelving operation information corresponding to the sample data set, wherein the shelving operation information includes a shelving person, a shelving time, a storage path, and a sample version number.