Training corpus quality evaluation method, device and equipment based on large model and storage medium

Abstract

The invention discloses a training corpus quality evaluation method, device and equipment based on a large model and a storage medium, is applied to an evaluation system architecture comprising a data set loading center, a file preprocessing center, a file pre-check center and a file quality inspection center, and relates to the technical field of artificial intelligence. Comprising the following steps: loading a data set directory to be subjected to quality inspection in a preset file database based on a preset quality inspection mode, and querying data files in the data set directory to be subjected to quality inspection based on a preset file path; processing each data file to obtain to-be-processed files, and packaging each to-be-processed file into a file list; pre-filtering each file to be processed in the file list, judging whether the file to be processed needs format conversion or not, and if so, converting the format of the file to be processed to obtain a file to be subjected to quality inspection; and performing file quality inspection on the file to be subjected to quality inspection by using the file quality inspector to obtain a quality inspection result, and evaluating the quality inspection result to obtain an evaluation result. In this way, the speed of quality evaluation of training corpora can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and particularly relates to a method, device, equipment and storage medium for evaluating the quality of training corpus based on a large model. Background Art

[0002] The large model corpus refers to a series of text, speech, video, pictures and other different modal data used for training and evaluating a large model, and the scale and quality of the corpus have a crucial impact on the performance of the large model and the depth and breadth of its applications.

[0003] Currently, there are problems such as incomplete coverage, insufficient accuracy, and insufficient timeliness in the large model training corpus, resulting in the large model usually being difficult to achieve the expected goal. Currently, in order to evaluate the quality of the training corpus, two technical solutions are usually adopted. One is to construct a binary classification model using high-quality data and low-quality data, and the other is to calculate the perplexity of the corpus based on a pre-trained model, and then set a certain perplexity threshold to screen high-quality data. However, the above two solutions are usually only applicable to quality inspection of data of a single file type or structured data after parsing the original file, and are not applicable to data of diverse file formats. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and storage medium for evaluating the quality of training corpus based on a large model, which can improve the speed and quality of evaluating the quality of the training corpus of the large model during the process of evaluating the quality of the training corpus, and thus accelerate the speed of the production process. The specific solutions are as follows:

[0005] In a first aspect, the present application provides a method for evaluating the quality of training corpus based on a large model, which is applied to an evaluation system architecture including a dataset loading center, a file preprocessing center, a file pre-inspection center and a file quality inspection center. The method includes:

[0006] Using the dataset loading center and based on a preset quality inspection mode, load the directory of the dataset to be quality inspected in a preset file database, and query the data files in the directory of the dataset to be quality inspected based on a preset file path; the data files include the corpus for training the large model;

[0007] Process each of the data files through the file preprocessing center to obtain corresponding files to be processed, and encapsulate each of the files to be processed into a file list; the file to be processed includes a file name, a file description, a dataset name to which the file belongs, a dataset description information to which the file belongs, a file format, a file size, and a file path;

[0008] Based on a preset filtering operation and using the file pre-inspection center to pre-filter each of the to-be-processed files in the file list, and determine whether the to-be-processed file needs format conversion. If so, perform format conversion on the to-be-processed file to obtain a file to be quality-inspected;

[0009] Use the file quality inspector in the file quality inspection center to perform file quality inspection operations on the file to be quality-inspected, obtain a quality inspection result, and use a preset quality assessment rule to evaluate the quality inspection result to obtain an evaluation result.

[0010] Optionally, the step of using the dataset loading center to load a directory of the dataset to be quality-inspected in a preset file database based on a preset quality inspection mode and querying data files in the directory of the dataset to be quality-inspected based on a preset file path includes:

[0011] Judge whether the quality inspection mode is a full-inspection mode. If the quality inspection mode is the full-inspection mode, load the directory of the dataset to be quality-inspected in the preset file database; if the quality inspection mode is not the full-inspection mode, load the directory of the dataset to be quality-inspected using a preset configuration file; the directory of the dataset to be quality-inspected includes a dataset name, the industry to which the dataset belongs, and a dataset introduction;

[0012] Traverse the directory of the dataset to be quality-inspected in sequence to query data files in each dataset in the directory of the dataset to be quality-inspected based on information of the file to be queried; the information of the file to be queried includes a file name, a file introduction, a file format, and a file path.

[0013] Optionally, the step of processing each of the data files through the file preprocessing center to obtain corresponding to-be-processed files and encapsulating each of the to-be-processed files into a file list includes:

[0014] Based on each of the to-be-processed files, establish corresponding temporary directories respectively, download the corresponding to-be-processed files based on the distributed file storage paths corresponding to each of the to-be-processed files, and store the to-be-processed files in the temporary directories;

[0015] Judge whether the to-be-processed file is a compressed file. If the to-be-processed file is a compressed file, unzip the to-be-processed file, and judge whether the currently unzipped file contains a compressed file;

[0016] If the currently unzipped file contains a compressed file, unzip the currently unzipped file to obtain a new currently unzipped file, and jump to the step of judging whether the currently unzipped file contains a compressed file until the currently unzipped file does not contain a compressed file;

[0017] Store the currently decompressed files in the temporary directory, traverse the files in the temporary directory, and encapsulate the traversed files into a file list.

[0018] Optionally, pre-filter each of the to-be-processed files in the file list based on a preset filtering operation and using the file pre-inspection center, and determine whether the to-be-processed files need format conversion. If so, perform format conversion on the to-be-processed files to obtain files to be quality inspected, including:

[0019] Use the file pre-inspection center to determine whether the dataset industry to which the files corresponding to each of the to-be-processed files in the file list belong is a preset standard industry. If the dataset industry to which the files corresponding to each of the to-be-processed files in the file list belong is a preset standard industry, use a preset file format filter to perform a file format verification operation on each of the to-be-processed files in the file list to determine whether the to-be-processed files are in a preset file format;

[0020] If the to-be-processed file is in the preset file format, call a preset file size filter to verify the file size corresponding to the to-be-processed file; the preset file format includes a text file format, a table file format, and a compressed file format;

[0021] If the file size is greater than a preset file size threshold, call a preset file name filter to verify the file name corresponding to the to-be-processed file. If the file name does not belong to a preset configuration shielding list, update the current number of quality inspection files;

[0022] Determine whether the current number of quality inspection files is greater than a preset quality inspection file number threshold. If the current number of quality inspection files is not greater than the preset quality inspection file number threshold, determine whether the file format corresponding to the to-be-processed file needs to be converted;

[0023] If the file format corresponding to the to-be-processed file needs to be converted, call a preset file format conversion service to perform a file format conversion operation on the to-be-processed file to obtain a file to be quality inspected and a new file path corresponding to the file to be quality inspected, and replace the original file path corresponding to the file to be quality inspected with the new file path.

[0024] Optionally, perform a file quality inspection operation on the file to be quality inspected using a file quality inspector in the file quality inspection center to obtain a quality inspection result, including:

[0025] Traverse the list of quality inspectors in the file quality inspection center based on user requirements to obtain the corresponding file quality inspectors, and then use the file quality inspectors to call the preset quality inspection method to perform quality inspection operations on each of the files to be quality inspected, so as to obtain the quality inspection dimension scores corresponding to each of the files to be quality inspected; the quality inspection dimension scores include the name of the quality inspector and the corresponding quality inspection score.

[0026] Optionally, the step of using the file quality inspector to call the preset quality inspection method to perform quality inspection operations on each of the files to be quality inspected, so as to obtain the quality inspection dimension scores corresponding to each of the files to be quality inspected, includes:

[0027] Determine the file format corresponding to the file to be quality inspected. If the file format corresponding to the file to be quality inspected is a text file format, use a preset encoding detection tool to detect the file encoding corresponding to the file to be quality inspected;

[0028] Read the file content of the file to be quality inspected through a preset text loading tool and determine whether the reading is successful. If the reading is not successful, read the file to be quality inspected based on the file encoding, and use a preset text sharding tool to shard the read file content to obtain a first sharding result, and then use a preset file scoring determiner to perform a correlation scoring determination operation on each of the first sharding results to obtain the corresponding quality inspection dimension score;

[0029] If the file format corresponding to the file to be quality inspected is a table file format, use a preset table processing tool to read the file content of the file to be quality inspected, and intercept the data in the first preset number of rows of the file content;

[0030] Convert the data format of the data in the preset number of rows to a structured format to obtain the corresponding structured data, and use a preset sharding tool to shard the structured data to obtain a second sharding result, and use the preset file scoring determiner to perform a correlation scoring determination operation on each of the second sharding results to obtain the corresponding quality inspection dimension score;

[0031] If the file format corresponding to the file to be quality inspected is a compressed file format, decompress the file to be quality inspected, and parse the file content corresponding to the decompressed file to obtain the corresponding parsing result, and use the preset file scoring determiner to perform a correlation scoring determination operation on each of the parsing results to obtain the corresponding quality inspection dimension score.

[0032] Optionally, the step of using a preset file scoring determiner to perform a correlation scoring determination operation on each of the first sharding results to obtain the corresponding quality inspection dimension score, includes:

[0033] Determine the quality inspection type corresponding to the file to be quality inspected. If the quality inspection type corresponding to the file to be quality inspected is a relevant quality inspection type, determine the corresponding first file processing operation based on the file type corresponding to the file to be quality inspected;

[0034] Invoke the first file parser corresponding to the file to be quality inspected and parse the first file content corresponding to the file to be quality inspected based on the first file processing operation, then construct a corresponding prompt word based on the first file content to perform a relevance judgment based on the prompt word, obtain a judgment result, and calculate the corresponding relevant quality inspection score based on the judgment result;

[0035] If the quality inspection type corresponding to the file to be quality inspected is an immediate quality inspection, determine the corresponding second file processing operation based on the file type corresponding to the file to be quality inspected, invoke the second file parser corresponding to the file to be quality inspected and parse the second file content corresponding to the file to be quality inspected based on the second file processing operation, extract the corresponding file release time from the second file content, and determine the corresponding immediate quality inspection score based on the preset timeliness determination rule and the file release time.

[0036] In a second aspect, the present application provides a training corpus quality evaluation device based on a large model, which is applied to an evaluation system architecture including a data set loading center, a file preprocessing center, a file pre-inspection center, and a file quality inspection center. The device includes:

[0037] A data file query module, configured to use the data set loading center and load a directory of data sets to be quality inspected in a preset file database based on a preset quality inspection mode, and query data files in the directory of data sets to be quality inspected based on a preset file path; the data files include corpora for training the large model;

[0038] A file encapsulation module, configured to process each of the data files through the file preprocessing center to obtain corresponding files to be processed, and encapsulate each of the files to be processed into a file list; the files to be processed include file names, file descriptions, data set names to which the files belong, data set description information to which the files belong, file formats, file sizes, and file paths;

[0039] A file pre-filtering module, configured to perform pre-filtering on each of the files to be processed in the file list based on a preset filtering operation and using the file pre-inspection center, and determine whether the files to be processed need to be format-converted. If so, perform format conversion on the files to be processed to obtain files to be quality inspected;

[0040] An evaluation result determination module, configured to perform a file quality inspection operation on the file to be quality inspected by using a file quality inspector in the file quality inspection center, obtain a quality inspection result, and evaluate the quality inspection result by using a preset quality evaluation rule to obtain an evaluation result.

[0041] In a third aspect, the present application provides an electronic device, including:

[0042] A memory, configured to store a computer program;

[0043] A processor, configured to execute the computer program to implement the foregoing method for evaluating the quality of training corpora based on a large model.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program, wherein when the computer program is executed by a processor, the foregoing method for evaluating the quality of training corpora based on a large model is implemented.

[0045] As can be seen from the above, before the present application performs the quality evaluation of the training corpora based on the large model, it is necessary to use the dataset loading center to load the directory of the dataset to be quality inspected in the preset file database based on the preset quality inspection mode, and query the data files in the directory of the dataset to be quality inspected based on the preset file path; the data files include the corpora for training the large model; each data file is processed by the file preprocessing center to obtain a corresponding file to be processed, and each file to be processed is encapsulated into a file list; the file to be processed includes a file name, a file description, a name of the dataset to which the file belongs, a description information of the dataset to which the file belongs, a file format, a file size, and a file path; each file to be processed in the file list is pre-filtered by using the file pre-inspection center based on the preset filtering operation, and it is determined whether the file to be processed needs to be format-converted. If so, the file to be quality inspected is format-converted to obtain the file to be quality inspected; a file quality inspection operation is performed on the file to be quality inspected by using a file quality inspector in the file quality inspection center to obtain a quality inspection result, and the quality inspection result is evaluated by using a preset quality evaluation rule to obtain an evaluation result.

[0046] It can be seen that, first, this application needs to use the dataset loading center to load the directory of the dataset to be quality inspected in the preset file database based on the preset quality inspection mode, and query the data files in the directory of the dataset to be quality inspected based on the preset file path; the data files include the corpus for training the large model; subsequently, the file preprocessing center processes each data file to obtain the corresponding file to be processed, and encapsulates each file to be processed into a file list; the file to be processed includes the file name, file description, name of the dataset to which the file belongs, description information of the dataset to which the file belongs, file format, file size, and file path; furthermore, based on the preset filtering operation, the file pre-inspection center pre-filters each file to be processed in the file list, and determines whether the file to be processed needs to be format-converted. If so, the file to be quality inspected is format-converted to obtain the file to be quality inspected; finally, the file quality inspector in the file quality inspection center performs a file quality inspection operation on the file to be quality inspected to obtain a quality inspection result, so as to evaluate the quality inspection result using the preset quality evaluation rule to obtain an evaluation result. In this way, the speed and quality of the quality evaluation of the training corpus of the large model are improved, thereby accelerating the speed of the production process and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative efforts.

[0048] Figure 1 It is a flowchart of a method for evaluating the quality of the training corpus based on a large model disclosed in this application;

[0049] Figure 2 It is a schematic flowchart of the architecture of a specific evaluation system disclosed in this application;

[0050] Figure 3 It is a schematic flowchart of a specific method for quality evaluation disclosed in this application;

[0051] Figure 4 It is a flowchart of a specific method for preprocessing files disclosed in this application;

[0052] Figure 5 It is a flowchart of a specific method for pre-inspecting files disclosed in this application;

[0053] Figure 6 It is a flowchart of a specific method for quality inspecting files disclosed in this application;

[0054] Figure 7A specific flowchart for file parsing using a CSV parser disclosed in this application;

[0055] Figure 8 A specific flowchart for file parsing using a docx parser disclosed in this application;

[0056] Figure 9 A specific flowchart for file parsing using a txt parser disclosed in this application;

[0057] Figure 10 A specific flowchart for file parsing using a json parser disclosed in this application;

[0058] Figure 11 A specific flowchart for file parsing using a parquet parser disclosed in this application;

[0059] Figure 12 A specific flowchart for file parsing using a jsonl parser disclosed in this application;

[0060] Figure 13 A specific schematic diagram of a correlation determination Prompt disclosed in this application;

[0061] Figure 14 A specific schematic diagram of an excel header row inference process disclosed in this application;

[0062] Figure 15 A specific schematic diagram of a statistical data discrimination process disclosed in this application;

[0063] Figure 16 A specific example diagram of a statistical data determination Prompt disclosed in this application;

[0064] Figure 17 A specific schematic diagram of a numerical type discrimination process disclosed in this application;

[0065] Figure 18 A specific example diagram of a timeliness time extraction Prompt disclosed in this application;

[0066] Figure 19 A specific example diagram of a data date extraction Prompt for statistical types and numerical types disclosed in this application;

[0067] Figure 20 A specific example diagram of a date column extraction Prompt based on an excel title column disclosed in this application;

[0068] Figure 21Schematic diagram of the structure of a training corpus quality evaluation device based on a large model disclosed in this application;

[0069] Figure 22 Structural diagram of an electronic device disclosed in this application. Detailed implementation manners

[0070] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0071] Currently, there are problems such as incomplete coverage, insufficient accuracy, and insufficient timeliness in the training corpus of large models, resulting in the fact that large models usually fail to achieve the expected goals. Currently, in order to evaluate the quality of the training corpus, two technical solutions are usually adopted. One is to construct a binary classification model using high-quality data and low-quality data, and the other is to calculate the perplexity of the corpus based on a pre-trained model, and then set a certain perplexity threshold to screen high-quality data. However, the above two solutions are usually only applicable to the quality inspection of data of a single file type or structured data after parsing the original file, and are not applicable to data of various file formats. Therefore, this application provides a method for evaluating the quality of the training corpus of a large model, which can improve the speed and quality of evaluating the quality of the training corpus of a large model, and thus accelerate the speed of the production process.

[0072] See Figure 1 As shown, an embodiment of the present invention discloses a method for evaluating the quality of a training corpus of a large model, which is applied to an evaluation system architecture including a dataset loading center, a file preprocessing center, a file pre-inspection center, and a file quality inspection center. The method includes:

[0073] Step S11: Use the dataset loading center to load the directory of the dataset to be quality-inspected in the preset file database based on a preset quality inspection mode, and query the data files in the directory of the dataset to be quality-inspected based on a preset file path; the data files include the corpus for training the large model.

[0074] In this embodiment, the evaluation system architecture for evaluating the quality of the training corpus based on a large model is as Figure 2 shown, including a dataset loading center, a file preprocessing center, a file pre-inspection center, and a file quality inspection center. The above centers include databases and provide HDFS (Hadoop Distributed File System, that is, a distributed file system) file storage functions and file conversion services.

[0075] Further, the schematic diagram of the process for evaluating the quality of training corpus using a large model is as Figure 3 shown: First, determine the current quality inspection mode. If the current quality inspection mode is the full inspection mode, load the directory of the dataset to be quality inspected from the database. If the current quality inspection mode is not the full inspection mode, load the directory of the quality inspection dataset from the configuration file. The directory of the quality inspection dataset contains metadata such as the dataset name, the industry to which the dataset belongs, and the dataset introduction. Then, traverse the directory of the quality inspection dataset in sequence, query the files included in each dataset, and the file information queried includes metadata such as the file name, file introduction, file format, and file path. Subsequently, obtain the HDFS path corresponding to the file, and through the file preprocessing process, implement the download and encapsulation of the file. The result returned after the file preprocessing process is a file list. Then, determine whether the file is a compressed file. If the file is not a compressed file, the file list contains only one element. If the file is a compressed file, the obtained file list is all the files after decompression. It is worth mentioning that each element of the file list is a dictionary, and the key values of the dictionary are the metadata information of the file, where the file path of the metadata is the absolute path where the file is downloaded to the local.

[0076] Specifically, using the dataset loading center and based on the preset quality inspection mode, loading the directory of the dataset to be quality inspected in the preset file database, and querying the data files in the directory of the dataset to be quality inspected based on the preset file path may include: determining whether the quality inspection mode is the full inspection mode. If the quality inspection mode is the full inspection mode, load the directory of the dataset to be quality inspected in the preset file database; if the quality inspection mode is not the full inspection mode, use the preset configuration file to load the directory of the dataset to be quality inspected; the directory of the dataset to be quality inspected includes the dataset name, the industry to which the dataset belongs, and the dataset introduction; traverse the directory of the dataset to be quality inspected in sequence to query the data files in each dataset in the directory of the dataset to be quality inspected based on the file information to be queried; the file information to be queried includes the file name, file introduction, file format, and file path.

[0077] Step S12, process each of the data files through the file preprocessing center to obtain corresponding files to be processed, and encapsulate each of the files to be processed into a file list; the file to be processed includes the file name, file description, name of the dataset to which the file belongs, description information of the dataset to which the file belongs, file format, file size, and file path.

[0078] In this embodiment, after using the dataset loading center and based on the preset quality inspection mode, loading the directory of the dataset to be quality inspected in the preset file database, and querying the data files in the directory of the dataset to be quality inspected based on the preset file path, the embodiments of the present application need to process each data file through the file preprocessing center. The schematic diagram of the process for preprocessing the file is as Figure 4Shown: In a specific embodiment, first, initialize a temporary directory for each file, and the name of the directory is UUID (Universally Unique Identifier), and then, according to the path of the file in hdfs, download the file to the local temporary directory. The above file download function can be implemented by using the krbcontext (i.e., a context) and hdfs library of Python (i.e., a programming language). Then, determine whether the current file is a compressed file. If it is a compressed file, decompress the downloaded file, and determine whether the decompressed file still contains a compressed file. If it still contains a compressed file, perform decompression operations on the compressed files in the current file in turn until there are no more compressed files. Finally, traverse the above temporary directory and encapsulate the files in the directory into a file list. Among them, the file information in the file list includes file path, file size, file name, file introduction, file format, etc. If the file is a decompressed file, the file introduction is the file introduction of the file before decompression.

[0079] Specifically, the file preprocessing center processes each data file to obtain corresponding files to be processed, and encapsulates each file to be processed into a file list, which may include: creating corresponding temporary directories based on each file to be processed respectively, downloading the corresponding files to be processed based on the distributed file storage paths corresponding to each file to be processed respectively, and storing the files to be processed in the temporary directories; determining whether the file to be processed is a compressed file. If the file to be processed is a compressed file, decompress the file to be processed, and determine whether the currently decompressed file contains a compressed file; if the currently decompressed file contains a compressed file, decompress the currently decompressed file to obtain a new currently decompressed file, and jump to the step of determining whether the currently decompressed file contains a compressed file until the currently decompressed file does not contain a compressed file; store the currently decompressed file in the temporary directory, traverse the files in the temporary directory, and encapsulate the traversed files into a file list.

[0080] Step S13: Based on a preset filtering operation and using the file pre-inspection center, pre-filter each file to be processed in the file list, and determine whether the file to be processed needs to be format-converted. If so, perform format conversion on the file to be processed to obtain a file to be quality-inspected.

[0081] In this embodiment, after the file preprocessing center processes each data file and encapsulates the obtained files to be processed into a file list, the embodiment of the present application needs to perform a pre-inspection operation on the files to be processed in the file list, and the schematic flow chart of pre-inspecting the files is as Figure 5Shown as follows: First, determine whether the dataset industry corresponding to the file to be processed is a standard industry. If the dataset industry corresponding to the file to be processed is not a standard industry, then end the operation of quality inspection on the current file. Otherwise, sequentially traverse the file pre-inspection filter list to respectively verify the file format, file size, and file name of the current file. Among them, the file format filter is used to verify the file format to determine whether the file is a supported format. The supported file formats include, but are not limited to, csv (Comma-Separated Values, that is, comma-separated values), xls (Excel Spreadsheet, that is, Excel spreadsheet), xlsx (Excel Open XML Spreadsheet, that is, Excel Open XML spreadsheet), txt (Text File, that is, text file), doc (Document File, that is, document file), docx (Document Open XML, that is, document Open XML), and json (JavaScript Object Notation, that is, JavaScript object notation). If the file is not a supported format, then end the operation of quality inspection on the current file. Among them, the file size filter is used to verify the file size. If the file size is less than the configured file size threshold, then end the quality inspection of the current file. That is to say, filtering out small files that do not contain valid content can improve the quality inspection efficiency. The file name filter verifies the file name. Further, if the file name belongs to the configured shielding list, then end the quality inspection of the current file.

[0082] It is worth mentioning that after the current file passes the verification of all filters, the embodiment of the present application needs to determine whether the number of quality inspection files in the current dataset exceeds the maximum number of quality inspection files limit. If it exceeds, then stop the operation of quality inspection on the current file. If it does not exceed, then determine whether the size of the quality inspection files in the current dataset exceeds the maximum quality inspection file size threshold. If it exceeds, then stop the operation of quality inspection on the current file. If it does not exceed, then increment the number of quality inspection files in the current dataset, and add the size of the current file to the size of the quality inspection files in the current dataset as the size of the quality inspection files in the current dataset. Finally, determine whether the format of the current file needs to be converted. If so, use the file format conversion service to convert the file format, and replace the local path of the current file with the local path of the file after format conversion. In a specific implementation manner, the embodiment of the present application can convert a doc format file to a docx format file, and the file format conversion service can be LibreOffice (that is, an open-source office software).

[0083] Specifically, based on a preset filtering operation and using a file pre-inspection center to pre-filter each to-be-processed file in a file list, and determine whether the to-be-processed file needs to be format-converted. If so, perform a format conversion on the to-be-processed file to obtain a file to be quality-inspected, which may include: using the file pre-inspection center to determine whether the industry of the data set to which each to-be-processed file in the file list belongs is a preset standard industry. If the industry of the data set to which each to-be-processed file in the file list belongs is a preset standard industry, use a preset file format filter to perform a file format verification operation on each to-be-processed file in the file list to determine whether the to-be-processed file is in a preset file format; if the to-be-processed file is in a preset file format, call a preset file size filter to verify the file size corresponding to the to-be-processed file; the preset file formats include a text file format, a table file format, and a compressed file format; if the file size is greater than a preset file size threshold, call a preset file name filter to verify the file name corresponding to the to-be-processed file. If the file name does not belong to a preset configuration shielding list, update the current number of quality-inspected files; determine whether the current number of quality-inspected files is greater than a preset quality-inspected file number threshold. If the current number of quality-inspected files is not greater than the preset quality-inspected file number threshold, determine whether the file format corresponding to the to-be-processed file needs to be converted; if the file format corresponding to the to-be-processed file needs to be converted, call a preset file format conversion service to perform a file format conversion operation on the to-be-processed file to obtain a file to be quality-inspected and a new file path corresponding to the file to be quality-inspected, and replace the original file path corresponding to the file to be quality-inspected with the new file path.

[0084] Step S14: Use a file quality inspector in the file quality inspection center to perform a file quality inspection operation on the file to be quality-inspected to obtain a quality inspection result, and use a preset quality evaluation rule to evaluate the quality inspection result to obtain an evaluation result.

[0085] In this embodiment, after using the file pre-inspection center to perform a pre-filtering operation on each to-be-processed file in the file list, the embodiment of the present application needs to perform a quality inspection on the file to be quality-inspected, and the flow diagram of the quality inspection of the file is as Figure 6 shown: First, initialize a quality inspection result object. Among them, the quality inspection result object includes quality inspection file metadata and scores of each quality inspection dimension. The key corresponding to each quality inspection dimension of the quality inspection object is the quality inspector name, and the value is the quality inspection score of the corresponding dimension. Subsequently, traverse the quality inspector list in sequence. Each element in the quality inspector list corresponds to a quality inspector corresponding to each quality inspection dimension, and use the quality inspection file metadata as input. At the same time, call the check method of the quality inspector to obtain a return result. The return result is the score of the file in this quality inspection dimension. The check method of the quality inspector is a method of parsing the file using a corresponding file parser selected according to the file format.

[0086] Specifically, a file quality checker in the file quality inspection center is used to perform file quality inspection operations on the files to be inspected, and the quality inspection results can be obtained, including: traversing the quality checker list in the file quality inspection center based on user requirements to obtain the corresponding file quality checker, and then using the file quality checker and calling the preset quality inspection method to perform quality inspection operations on each file to be inspected respectively to obtain the quality inspection dimension scores corresponding to each file to be inspected; the quality inspection dimension scores include the name of the quality checker and the corresponding quality inspection scores.

[0087] Further, in a specific implementation manner, the schematic flow diagram of using a CSV parser for file parsing is as Figure 7 shown: First, use uchardet to detect the file encoding, and then, through Pandas (i.e., a Python data analysis library), read the CSV file using the default encoding rule. If the file reading fails, then use Pandas to read the CSV file using the detected file encoding. In addition, when using Pandas to load the CSV file, the embodiments of the present application can select the amount of data to be loaded according to user requirements.

[0088] Among them, in a specific implementation manner, the schematic flow diagram of using a docx parser for file parsing is as Figure 8 shown: Use the docx library of Python to load the docx document, and use xpath to match all text tags, and then splice the text of all tags with an empty string; subsequently, use the RecursiveCharacterTextSplitter to process the parsed text fragments, that is, use docx to parse the header and footer of the docx document to retain non-empty values and remove duplicates, and then return the parsed docx document fragments, pages, and footers as the docx file parsing result.

[0089] In a specific implementation manner, the schematic flow diagram of using a txt parser for file parsing is as Figure 9 shown: First, use uchardet to detect the file encoding, and read the txt file using the automatically detected encoding through a TextLoader (i.e., a text loader), and judge the file reading status. If the reading fails, then use the TextLoader to read the txt file based on the detected file encoding. Subsequently, use the RecursiveCharacterTextSplitter (i.e., the recursive character text splitter) to fragment the content of the parsed txt file, and return the fragmentation result as the txt file parsing result.

[0090] In a specific implementation manner, the schematic flow diagram of using a json parser for file parsing is as Figure 10Shown as follows: First, use the JSON module to read the JSON file and determine whether the parsed content is a JSON array. If so, sequentially traverse the first N elements of the JSON array, use RecursiveJsonSplitter for slicing, and then splice the obtained element slicing results into an array. If the parsed content is not a JSON array, use RecursiveJsonSplitter to slice the obtained parsed content, and intercept the first N slices of the obtained JSON file slice array as the json file parsing result.

[0091] In a specific implementation, the process schematic diagram of using a parquet (i.e., columnar storage format) parser for file parsing is as Figure 11 Shown as follows: First, use Pandas to read the parquet file as a DataFrame, and intercept the first N rows of the DataFrame. Then, convert the intercepted DataFrame into a JSON array. Subsequently, sequentially traverse the JSON array, use RecursiveJsonSplitter to slice each element in the JSON array, and then take the first N slices of the intercepted JSON slices as the parquet file parsing result.

[0092] In a specific implementation, the process schematic diagram of using a jsonl (JSON Lines, i.e., JSON line format) parser for file parsing is as Figure 12Shown as follows: First, use the open method to read the jsonl file, and sequentially traverse the first N lines of data. Then, use json to parse the content read in each line, and use RecursiveJsonSplitter to slice the parsing result. Subsequently, splice the sliced results of each line into an array, and use the first N slices obtained by intercepting the JSONL slices as the parsing result of the JSONL file. Specifically, use a file quality inspector and call a preset quality inspection method to perform quality inspection operations on each file to be quality inspected respectively, so as to obtain the quality inspection dimension scores corresponding to each file to be quality inspected, which may include: determining the file format corresponding to the file to be quality inspected. If the file format corresponding to the file to be quality inspected is a text file format, use a preset encoding detection tool to detect the file encoding corresponding to the file to be quality inspected; read the file content of the file to be quality inspected through a preset text loading tool, and determine whether the reading is successful. If the reading is not successful, read the file to be quality inspected based on the file encoding, and use a preset text slicing tool to slice the read file content to obtain a first slicing result. Then, use a preset file score determiner to perform a relevance score determination operation on each first slicing result to obtain the corresponding quality inspection dimension score; if the file format corresponding to the file to be quality inspected is a table file format, use a preset table processing tool to read the file content of the file to be quality inspected, and intercept the data in the first preset number of rows of the file content; convert the data format of the data in the preset number of rows into a structured format to obtain the corresponding structured data, and use a preset slicing tool to slice the structured data to obtain a second slicing result, and use a preset file score determiner to perform a relevance score determination operation on each second slicing result to obtain the corresponding quality inspection dimension score; if the file format corresponding to the file to be quality inspected is a compressed file format, decompress the file to be quality inspected, and parse the file content corresponding to the decompressed file to obtain the corresponding parsing result, and use a preset file score determiner to perform a relevance score determination operation on each parsing result to obtain the corresponding quality inspection dimension score.

[0093] It is worth mentioning that in a specific implementation manner, the process of performing quality inspection using the check method in a relevant quality inspector is as follows: First, determine the file type. If the file is in docx file format or txt file format, call the file parser corresponding to the file format respectively, and then read the first slice of the parsing result to construct a large model Prompt (i.e., a prompt for command line interaction) for relevance judgment. Among them, if the large model result indicates relevance, set the relevance score to 2 points. If the large model result is uncertain, set the relevance score to 1 point. If it indicates no relevance, set the relevance score to 0 point.

[0094] In addition, if the file format is json, jsonl, or parquet, the corresponding file parsers for these file formats are called respectively. Then, all shards are traversed in sequence to construct a large model Prompt for relevance judgment, and the average value of the relevance scores of all shards is calculated. The obtained average value is used as the relevance score of the entire file.

[0095] If the file format is csv, the corresponding file parser for this file format is called to load the file. Then, it is determined whether the current file is statistical data or numerical type data. If the current file is statistical data or numerical type data, the first N rows of the csv file are read, and then the large model Prompt is used for relevance judgment. If the current file is not statistical data or numerical type data, the first N rows are traversed in sequence to construct a large model Prompt for relevance judgment, and the average value of the relevance scores of all rows is calculated as the relevance score of the entire file.

[0096] If the file format is xls or xlsx, first, the row index where the header row of the excel is located is inferred, and the excel (i.e., a Microsoft spreadsheet software) file is loaded using Pandas. Then, it is determined whether the current file is statistical data or numerical type data. If the current file is statistical data or numerical type data, the first N rows of the excel file are read, and the large model Prompt is used for relevance judgment. If the current file is not statistical data or numerical type data, the first N rows are traversed in sequence to construct a large model Prompt for relevance judgment, and the average value of the relevance scores of all rows is calculated as the relevance score of the entire file.

[0097] Specifically, using a preset file score determiner to perform an operation to determine the relevance score for each first shard result to obtain the corresponding quality inspection dimension score may include: determining the quality inspection type corresponding to the file to be quality inspected. If the quality inspection type corresponding to the file to be quality inspected is a relevance quality inspection type, then determining the corresponding first file processing operation based on the file type corresponding to the file to be quality inspected; calling the first file parser corresponding to the file to be quality inspected and parsing the first file content corresponding to the file to be quality inspected based on the first file processing operation, and then constructing a corresponding prompt word based on the first file content to perform a relevance judgment based on the prompt word to obtain a judgment result, and calculating the corresponding relevance quality inspection score based on the judgment result. Among them, in a specific implementation manner, a schematic diagram of an example of the Prompt for performing relevance determination is as Figure 13 shown.

[0098] If the quality inspection type corresponding to the file to be quality inspected is real-time quality inspection, determine the corresponding second file processing operation based on the file type corresponding to the file to be quality inspected, call the second file parser corresponding to the file to be quality inspected, parse the second file content corresponding to the file to be quality inspected based on the second file processing operation, extract the corresponding file release time from the second file content, and determine the corresponding real-time quality inspection score based on the preset timeliness determination rule and the file release time.

[0099] In a specific implementation, the process schematic diagram for inferring the excel header row is as Figure 14 shown: First, read the first N rows of data of the file corresponding to the xls file format or xlsx file format through Pandas, and set the header parameter to None. Subsequently, traverse each row of data in turn, count the number of string columns and the cumulative character length of the string columns in each row, calculate the ratio of the string columns and the average length of the string columns in each row according to the statistical results, and finally, find the string column whose row string column ratio is greater than the specified threshold, and use the row with the smallest average length in the string column as the header row.

[0100] Among them, the process schematic diagram for discriminating the statistical data is as Figure 15 shown: First, if the file format is the csv file format, use the CSV file parser to load the file. Otherwise, use Pandas to load the file. Subsequently, traverse each row of data in turn, intercept each column length according to the maximum length, and then splice each column of each row into an array to convert the data arrays of each column in the first N rows into strings, with each row corresponding to a line of the string. Then, construct a large model Prompt to use the large model to determine whether the excel is statistical data based on the first N rows of data. Among them, the schematic diagram of the Prompt example for determining the statistical data is as Figure 16 shown.

[0101] It is worth mentioning that the process schematic diagram for discriminating numerical type data is as Figure 17 shown: If the file format is the csv file format, use the CSV file parser to load the file. If the file format is not the csv file format, load the file through Pandas, and count the ratio of numerical cells and the eigenvalue of the average cell length. If the statistical result meets the corresponding statistical feature threshold, it is determined as numerical excel.

[0102] In a specific implementation, the process of quality inspection using the check method in the real-time quality inspector is as follows: First, determine the file type. If the file is a txt file, call the file parser for the corresponding file format, then read the first shard of the parsing result, and construct a large model Prompt to extract the publication time of the data. Then, perform a timeliness score based on the timeliness judgment rules, where within half a year is set to 3 points, 1 to 2 years is set to 2 points, 2 to 3 years is set to 1 point, and more than 3 years is set to 0 points. If the file format is one of json, jsonl, or parquet, call the corresponding file parser for parsing respectively, and then traverse all shards in sequence to construct a large model Prompt to extract the publication time of the shard data. Finally, determine the timeliness score of each shard according to the timeliness judgment rules, and calculate the average value of the timeliness scores of all shards as the timeliness score of the entire file. Among them, an example diagram of the Prompt for extracting the timeliness time is as Figure 18 shown.

[0103] Furthermore, if the file is in csv format, call the corresponding file parser to load the file. If the file is in xls or xlsx format, first infer the row index where the excel header row is located, and use Pandas to load the excel file. Then, determine whether the current file is statistical data or numerical type data. If the current file is statistical data or numerical type data, read the first N rows of the file, and then use the corresponding large model Prompt to extract the publication time. If the publication time is extracted, perform a timeliness score. If not, obtain the title column of the file, and then use the date column extraction Prompt. Furthermore, determine whether the file contains a date column. If it contains a date column, traverse the values of the date column of the file in sequence, and then calculate the average value of the timeliness scores of the first N rows of the date column as the timeliness score of the file. If it does not contain a date column, or the file is not statistical data or numerical type data, traverse the first N rows of the file in sequence, and use the content of each row as the Prompt input as a whole to extract the time using the time extraction Prompt, and perform a timeliness score based on the extracted time, and use the average value of the timeliness scores of the first N rows as the timeliness score of the file. Among them, an example diagram of the Prompt for date extraction for statistical type and numerical type data is as Figure 19 shown, and an example diagram of the Prompt for extracting the date column based on the excel title column is as Figure 20 shown.

[0104] In this embodiment, in a specific implementation, if the file format is a docx file, the file parser corresponding to the file format is called to read the first shard and the last shard in the parsing result, and the obtained first shard and the last shard are concatenated together to construct a large model Prompt. Subsequently, the release time of the data is extracted to perform a timeliness score according to the timeliness determination rule. Then, the header and footer of the file are obtained. If at least one of the header and footer is not empty, the header and footer are used to construct a large model to extract the data release time Prompt. Subsequently, a timeliness score is performed according to the timeliness determination rule. Finally, the maximum value among the timeliness scores of the file shards, the header, and the footer is used as the timeliness score of the docx file.

[0105] It can be seen that in the embodiment of the present application, it is first necessary to use the dataset loading center to load the directory of the dataset to be quality inspected in the preset file database based on the preset quality inspection mode, and query the data files in the directory of the dataset to be quality inspected based on the preset file path; the data files include the corpus for training the large model; subsequently, each data file is processed by the file preprocessing center to obtain the corresponding file to be processed, and each file to be processed is encapsulated into a file list; the file to be processed includes the file name, file description, name of the dataset to which the file belongs, description information of the dataset to which the file belongs, file format, file size, and file path; furthermore, based on the preset filtering operation, the file pre-inspection center is used to pre-filter each file to be processed in the file list, and it is judged whether the file to be processed needs to be format-converted. If so, the file to be quality inspected is format-converted to obtain the file to be quality inspected; finally, the file quality inspector in the file quality inspection center is used to perform a file quality inspection operation on the file to be quality inspected to obtain a quality inspection result, and the preset quality evaluation rule is used to evaluate the quality inspection result to obtain an evaluation result. In this way, the speed and quality of the quality evaluation of the training corpus of the large model are improved, and thus the speed of the production process is accelerated.

[0106] Correspondingly, as shown in Figure 21 the present application also provides a device for evaluating the quality of the training corpus of a large model, which is applied to an evaluation system architecture including a dataset loading center, a file preprocessing center, a file pre-inspection center, and a file quality inspection center. The device includes:

[0107] A data file query module 11, configured to use the dataset loading center to load a directory of a dataset to be quality inspected in a preset file database based on a preset quality inspection mode, and query data files in the directory of the dataset to be quality inspected based on a preset file path; the data files include the corpus for training the large model;

[0108] A file encapsulation module 12 is used to process each of the data files through the file preprocessing center to obtain corresponding files to be processed, and encapsulate each of the files to be processed into a file list; the file to be processed includes a file name, a file description, a name of the dataset to which the file belongs, a description information of the dataset to which the file belongs, a file format, a file size, and a file path.

[0109] A file pre-filtering module 13 is used to pre-filter each of the files to be processed in the file list based on a preset filtering operation and by using the file pre-inspection center, and determine whether the file to be processed needs to be format-converted. If so, the file to be processed is format-converted to obtain a file to be quality-inspected.

[0110] An evaluation result determination module 14 is used to perform a file quality inspection operation on the file to be quality-inspected by using a file quality inspector in the file quality inspection center to obtain a quality inspection result, and use a preset quality evaluation rule to evaluate the quality inspection result to obtain an evaluation result.

[0111] As can be seen from the above, before performing the quality evaluation of the training corpus based on the large model in the embodiment of the present application, it is first necessary to load the directory of the dataset to be quality-inspected in the preset file database by using the dataset loading center and based on the preset quality inspection mode, and query the data files in the directory of the dataset to be quality-inspected based on the preset file path; the data files include the corpus for training the large model; subsequently, each data file is processed through the file preprocessing center to obtain a corresponding file to be processed, and each file to be processed is encapsulated into a file list; the file to be processed includes a file name, a file description, a name of the dataset to which the file belongs, a description information of the dataset to which the file belongs, a file format, a file size, and a file path; furthermore, each file to be processed in the file list is pre-filtered based on the preset filtering operation and by using the file pre-inspection center, and it is determined whether the file to be processed needs to be format-converted. If so, the file to be quality-inspected is format-converted to obtain a file to be quality-inspected; finally, a file quality inspection operation is performed on the file to be quality-inspected by using the file quality inspector in the file quality inspection center to obtain a quality inspection result, and the quality inspection result is evaluated by using the preset quality evaluation rule to obtain an evaluation result. In this way, the speed and quality of the quality evaluation of the training corpus of the large model are improved, and thus the speed of the production process is accelerated.

[0112] In some specific embodiments, the data file query module 11 may specifically include:

[0113] The quality inspection mode judgment unit is used to judge whether the quality inspection mode is the full inspection mode. If the quality inspection mode is the full inspection mode, the directory of the dataset to be quality inspected is loaded in the preset file database; if the quality inspection mode is not the full inspection mode, the directory of the dataset to be quality inspected is loaded by using a preset configuration file; the directory of the dataset to be quality inspected includes the dataset name, the industry to which the dataset belongs, and the dataset introduction.

[0114] The dataset directory traversal unit is used to traverse the directory of the dataset to be quality inspected in sequence, so as to query data files in each dataset in the directory of the dataset to be quality inspected based on the information of the file to be queried; the information of the file to be queried includes the file name, the file introduction, the file format, and the file path.

[0115] In some specific embodiments, the file encapsulation module 12 may specifically include:

[0116] The file download unit is used to create a corresponding temporary directory for each of the files to be processed, download the corresponding files to be processed based on the distributed file storage paths corresponding to each of the files to be processed, and store the files to be processed in the temporary directory.

[0117] The file judgment unit is used to judge whether the file to be processed is a compressed file. If the file to be processed is a compressed file, the file to be processed is decompressed, and it is judged whether the decompressed file currently contains a compressed file.

[0118] The file decompression unit is used to, if the decompressed file currently contains a compressed file, decompress the decompressed file currently to obtain a new decompressed file currently, and jump to the step of judging whether the decompressed file currently contains a compressed file until the decompressed file currently does not contain a compressed file.

[0119] The file encapsulation word unit is used to store the decompressed file currently in the temporary directory, traverse the files in the temporary directory, and encapsulate the traversed files into a file list.

[0120] In some specific embodiments, the file pre-filtering module 13 may specifically include:

[0121] The file format verification unit is used to use the file pre-inspection center to judge whether the industry of the dataset to which the file corresponding to each of the files to be processed in the file list belongs is a preset standard industry. If the industry of the dataset to which the file corresponding to each of the files to be processed in the file list belongs is a preset standard industry, a preset file format filter is used to perform a file format verification operation on each of the files to be processed in the file list, so as to judge whether the file to be processed is a preset file format.

[0122] A file size verification unit, configured to, if the file to be processed is in the preset file format, call a preset file size filter to verify the file size corresponding to the file to be processed; the preset file format includes a text file format, a table file format, and a compressed file format;

[0123] A file name verification unit, configured to, if the file size is greater than a preset file size threshold, call a preset file name filter to verify the file name corresponding to the file to be processed, and if the file name does not belong to a preset configuration blacklist, update the current quality inspection file quantity;

[0124] A file quantity verification unit, configured to determine whether the current quality inspection file quantity is greater than a preset quality inspection file quantity threshold, and if the current quality inspection file quantity is not greater than the preset quality inspection file quantity threshold, determine whether the file format corresponding to the file to be processed needs to be converted;

[0125] A file path replacement unit, configured to, if the file format corresponding to the file to be processed needs to be converted, call a preset file format conversion service to perform a file format conversion operation on the file to be processed, obtain a file to be quality inspected and a new file path corresponding to the file to be quality inspected, and replace the original file path corresponding to the file to be quality inspected with the new file path.

[0126] In some specific embodiments, the evaluation result determination module 14 may specifically include:

[0127] A file quality inspector determination unit, configured to traverse the quality inspector list in the file quality inspection center based on user requirements to obtain a corresponding file quality inspector, and then use the file quality inspector and call a preset quality inspection method to perform quality inspection operations on each of the files to be quality inspected respectively, so as to obtain a quality inspection dimension score corresponding to each of the files to be quality inspected; the quality inspection dimension score includes a quality inspector name and a corresponding quality inspection score.

[0128] In some specific embodiments, the evaluation result determination module 14 may specifically include:

[0129] A file encoding determination unit, configured to determine the file format corresponding to the file to be quality inspected, and if the file format corresponding to the file to be quality inspected is a text file format, use a preset encoding detection tool to detect the file encoding corresponding to the file to be quality inspected;

[0130] A file content reading unit, configured to read the file content of the file to be quality inspected through a preset text loading tool, and determine whether the reading is successful. If the reading is not successful, the file to be quality inspected is read based on the file encoding, and the read file content is segmented by using a preset text segmentation tool to obtain a first segmentation result. Then, a preset file scoring determiner is used to perform a relevance scoring determination operation on each of the first segmentation results to obtain the corresponding quality inspection dimension score;

[0131] A data interception unit, configured to, if the file format of the file to be quality inspected is a table file format, read the file content of the file to be quality inspected by using a preset table processing tool, and intercept the data in the first preset number of rows of the file content;

[0132] A first quality inspection dimension score determination unit, configured to convert the data format of the data in the first preset number of rows into a structured format to obtain the corresponding structured data, segment the structured data by using a preset segmentation tool to obtain a second segmentation result, and use the preset file scoring determiner to perform a relevance scoring determination operation on each of the second segmentation results to obtain the corresponding quality inspection dimension score;

[0133] A second quality inspection dimension score determination unit, configured to, if the file format of the file to be quality inspected is a compressed file format, decompress the file to be quality inspected, parse the file content corresponding to the decompressed file to obtain the corresponding parsing result, and use the preset file scoring determiner to perform a relevance scoring determination operation on each of the parsing results to obtain the corresponding quality inspection dimension score.

[0134] In some specific embodiments, the evaluation result determination module 14 may specifically include:

[0135] A quality inspection type judgment unit, configured to determine the quality inspection type corresponding to the file to be quality inspected. If the quality inspection type corresponding to the file to be quality inspected is a relevant quality inspection type, a corresponding first file processing operation is determined based on the file type corresponding to the file to be quality inspected;

[0136] A relevant quality inspection score determination unit, configured to call a first file parser corresponding to the file to be quality inspected and parse the first file content corresponding to the file to be quality inspected based on the first file processing operation, then construct a corresponding prompt word based on the first file content, perform a relevance judgment based on the prompt word to obtain a judgment result, and calculate a corresponding relevant quality inspection score based on the judgment result;

[0137] A timely quality inspection score determination unit, which is used to, if the quality inspection type corresponding to the file to be quality inspected is timely quality inspection, determine a corresponding second file processing operation based on the file type corresponding to the file to be quality inspected, call a second file parser corresponding to the file to be quality inspected, parse the second file content corresponding to the file to be quality inspected based on the second file processing operation, extract the corresponding file release time from the second file content, and determine the corresponding timely quality inspection score based on a preset timeliness determination rule and the file release time.

[0138] Furthermore, an embodiment of the present application also discloses an electronic device. Figure 22 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure should not be considered as any limitation on the scope of use of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the method for evaluating the quality of training corpus based on a large model disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0139] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed on it here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application needs, and no specific limitation is made here.

[0140] In addition, as a carrier for resource storage, the memory 22 may be a read-only memory, a random access memory, a disk, or an optical disc, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be short-term storage or permanent storage.

[0141] Among them, the operating system 221 is used to manage and control each hardware device on the electronic device 20 and the computer program 222, and it may be Windows Server, Netware, Unix, Linux, etc. The computer program 222 may further include a computer program capable of completing other specific tasks in addition to the computer program capable of implementing the method for evaluating the quality of training corpus based on a large model executed by the electronic device 20 disclosed in any of the foregoing embodiments.

[0142] Further, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned method for evaluating the quality of training corpus based on a large model. For the specific steps of this method, reference may be made to the corresponding content disclosed in the foregoing embodiments, and details will not be elaborated herein.

[0143] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference may be made to each other. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and reference may be made to the description of the method part for related parts.

[0144] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0145] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0146] Finally, it should be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0147] The above has introduced the technical solution provided by this application in detail. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A training corpus quality assessment method based on a large model, characterized in that: Applied to an evaluation system architecture including a data set loading center, a file preprocessing center, a file pre-checking center, and a file quality inspection center, the method includes: Using the data set loading center and based on a preset quality inspection mode, loading a directory of data sets to be inspected in a preset file database, and querying data files in the directory of data sets to be inspected based on a preset file path; the data files include corpora used to train the large model; The file preprocessing center processes each of the data files to obtain a corresponding file to be processed, and encapsulates each of the files to be processed into a file list; the file to be processed includes a file name, a file description, a name of a data set to which the file belongs, description information of the data set to which the file belongs, a file format, a file size, and a file path; Based on the preset filtering operation and using the file pre-inspection center, each of the to-be-processed files in the file list is pre-filtered, and it is determined whether the to-be-processed files need to be format converted. If necessary, the to-be-processed files are format converted to obtain files to be quality inspected; The document quality checker in the document quality check center is used to perform a document quality check operation on the document to be quality checked to obtain a quality check result, and the quality check result is evaluated using a preset quality evaluation rule to obtain an evaluation result.

2. The method for evaluating the quality of training corpus based on a large model according to claim 1, characterized in that: The method of using the data set loading center to load the directory of the data set to be inspected in the preset file database based on the preset quality inspection mode, and querying the data files in the directory of the data set to be inspected based on the preset file path, includes: Determine whether the quality inspection mode is the full inspection mode. If the quality inspection mode is the full inspection mode, load the directory of the data set to be inspected in the preset file database; if the quality inspection mode is not the full inspection mode, load the directory of the data set to be inspected using a preset configuration file; the directory of the data set to be inspected includes the name of the data set, the industry to which the data set belongs, and a brief introduction to the data set; The directory of data sets to be inspected is traversed in sequence to search for data files in each data set in the directory of data sets to be inspected based on the file information to be queried; the file information to be queried includes file name, file introduction, file format and file path.

3. The method for evaluating the quality of training corpus based on a large model according to claim 1, characterized in that: The file preprocessing center processes each of the data files to obtain a corresponding file to be processed, and encapsulates each of the files to be processed into a file list, including: Establishing corresponding temporary directories based on the files to be processed respectively, downloading the corresponding files to be processed based on the distributed file storage paths corresponding to the files to be processed respectively, and storing the files to be processed in the temporary directories; Determine whether the file to be processed is a compressed file. If the file to be processed is a compressed file, decompress the file to be processed and determine whether the current decompressed file contains a compressed file. If the current decompressed file contains a compressed file, decompress the current decompressed file to obtain a new current decompressed file, and jump to the step of determining whether the current decompressed file contains a compressed file until the current decompressed file does not contain a compressed file; The currently decompressed file is stored in the temporary directory, and the files in the temporary directory are traversed, and the traversed files are encapsulated into a file list.

4. The method for evaluating the quality of training corpus based on a large model according to claim 1, characterized in that: The pre-filtering of each of the to-be-processed files in the file list based on the preset filtering operation and using the file pre-inspection center, and determining whether the to-be-processed files need to be format converted, and if necessary, performing format conversion on the to-be-processed files to obtain the to-be-processed files, includes: Using the file pre-check center to determine whether the industry of the file data set corresponding to each of the to-be-processed files in the file list is a preset standard industry; if the industry of the file data set corresponding to each of the to-be-processed files in the file list is a preset standard industry, using a preset file format filter to perform a file format verification operation on each of the to-be-processed files in the file list to determine whether the to-be-processed files are of the preset file format; If the file to be processed is in the preset file format, calling a preset file size filter to verify the file size corresponding to the file to be processed; the preset file format includes a text file format, a table file format and a compressed file format; If the file size is greater than the preset file size threshold, the preset file name filter is called to verify the file name corresponding to the file to be processed. If the file name does not belong to the preset configuration shielding list, the current quality inspection file quantity is updated; Determine whether the current number of quality inspection files is greater than a preset quality inspection file number threshold, and if the current number of quality inspection files is not greater than the preset quality inspection file number threshold, determine whether the file format corresponding to the to-be-processed file needs to be converted; If the file format corresponding to the file to be processed needs to be converted, the preset file format conversion service is called to perform file format conversion operation on the file to be processed, to obtain the file to be quality inspected and a new file path corresponding to the file to be quality inspected, and to replace the original file path corresponding to the file to be quality inspected with the new file path.

5. The method for evaluating the quality of training corpus based on a large model according to any one of claims 1 to 4, characterized in that: The using of the document quality inspector in the document quality inspection center to perform a document quality inspection operation on the document to be quality inspected to obtain a quality inspection result includes: Based on user needs, the list of quality inspectors in the file quality inspection center is traversed to obtain the corresponding file quality inspector, and then the file quality inspector is used to call the preset quality inspection method to perform quality inspection operations on each of the files to be quality inspected to obtain the quality inspection dimension score corresponding to each of the files to be quality inspected; the quality inspection dimension score includes the quality inspector name and the corresponding quality inspection score.

6. The method for evaluating the quality of training corpus based on a large model according to claim 5, characterized in that: The using the file quality checker and calling a preset quality check method to perform quality check operations on each of the files to be quality checked, so as to obtain a quality check dimension score corresponding to each of the files to be quality checked, includes: Determine the file format corresponding to the file to be inspected, and if the file format corresponding to the file to be inspected is a text file format, use a preset encoding detection tool to detect the file encoding corresponding to the file to be inspected; The file content of the file to be inspected is read by a preset text loading tool, and it is determined whether the reading is successful. If the reading is unsuccessful, the file to be inspected is read based on the file encoding, and the read file content is segmented by a preset text segmentation tool to obtain a first segmentation result, and then a preset file score determiner is used to perform a correlation score determination operation on each of the first segmentation results to obtain a corresponding quality inspection dimension score; If the file format corresponding to the file to be inspected is a table file format, a preset table processing tool is used to read the file content of the file to be inspected, and data in the first preset number of rows of the file content is intercepted; Convert the data format of the data in the preset number of rows into a structured format to obtain corresponding structured data, and use a preset sharding tool to shard the structured data to obtain second sharding results, and use the preset file score determiner to perform a relevance score determination operation on each of the second sharding results to obtain a corresponding quality inspection dimension score; If the file format corresponding to the file to be quality inspected is a compressed file format, the file to be quality inspected is decompressed, and the file content corresponding to the decompressed file is parsed to obtain the corresponding parsing result, and the preset file scoring determiner is used to perform a relevance scoring determination operation on each of the parsing results to obtain the corresponding quality inspection dimension score.

7. The method for evaluating the quality of training corpus based on a large model according to claim 6, characterized in that: The using a preset file score determiner to perform a correlation score determination operation on each of the first shard results to obtain a corresponding quality inspection dimension score includes: Determine a quality inspection type corresponding to the file to be inspected, and if the quality inspection type corresponding to the file to be inspected is a related quality inspection type, determine a corresponding first file processing operation based on the file type corresponding to the file to be inspected; Calling a first file parser corresponding to the file to be inspected and parsing a first file content corresponding to the file to be inspected based on the first file processing operation, then constructing a corresponding prompt word based on the first file content, performing a relevance judgment based on the prompt word, obtaining a judgment result, and calculating a corresponding relevance quality inspection score based on the judgment result; If the quality inspection type corresponding to the file to be quality inspected is timely quality inspection, a corresponding second file processing operation is determined based on the file type corresponding to the file to be quality inspected, and a second file parser corresponding to the file to be quality inspected is called and the second file content corresponding to the file to be quality inspected is parsed based on the second file processing operation, and the corresponding file release time is extracted from the second file content, and a corresponding timeliness quality inspection score is determined based on a preset timeliness determination rule and the file release time.

8. A training corpus quality assessment device based on a large model, characterized in that: Applicable to an evaluation system architecture including a data set loading center, a file preprocessing center, a file pre-inspection center, and a file quality inspection center, the device comprising: A data file query module, used to use the data set loading center and load the directory of the data set to be inspected in the preset file database based on the preset quality inspection mode, and query the data files in the directory of the data set to be inspected based on the preset file path; the data files include corpus used to train the large model; A file encapsulation module is used to process each of the data files through the file preprocessing center to obtain a corresponding file to be processed, and encapsulate each of the files to be processed into a file list; the file to be processed includes a file name, a file description, a name of a data set to which the file belongs, description information of the data set to which the file belongs, a file format, a file size, and a file path; A file pre-filtering module, used for pre-filtering each of the to-be-processed files in the file list based on a preset filtering operation and using the file pre-inspection center, and determining whether the to-be-processed files need to be format converted, and if so, performing format conversion on the to-be-processed files to obtain files to be quality inspected; The evaluation result determination module is used to use the file quality inspector in the file quality inspection center to perform a file quality inspection operation on the file to be quality inspected to obtain a quality inspection result, and to use preset quality evaluation rules to evaluate the quality inspection result to obtain an evaluation result.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the large model-based training corpus quality assessment method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the large model-based training corpus quality assessment method as described in any one of claims 1 to 7 is implemented.

Images