Method, device, electronic device and storage medium for constructing training data set

Abstract

The present invention provides a method, device, electronic device and storage medium for constructing a training data set, which belongs to the field of artificial intelligence technology, and includes: preprocessing a collected first data set to obtain a second data set, including converting non-text type data into text type data; obtaining a comprehensive classification score of each data in the second data set; based on the comprehensive classification score of each data in the second data set, screening out a target training data set from the second data set. The method, device, electronic device and storage medium for constructing a training data set provided by the present invention calculate the comprehensive classification score of each data by introducing an adaptive field evaluation function, and can dynamically evaluate and screen data according to various scenarios and field requirements, thereby having obvious technical improvement effects in broadening data sources, reducing cleaning costs, unifying quality standards and improving data purity in professional fields.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for constructing a training data set. Background Art

[0002] In the training process of large models, the pre-training of the model is particularly critical, accounting for more than 95% of the training time. In this process, the model builds a basic knowledge framework through large-scale data. The higher the quality of the data used for model training, the stronger the model's cognitive and generalization capabilities.

[0003] However, the training data currently used for pre-training mainly relies on web crawling. Although the amount is huge, there are generally problems such as uneven quality and lack of professionalism. Taking an autoregressive language model as an example, the high-quality training data used to train it accounts for less than 20%, and is scattered in different scenarios and fields. The proportion of high-quality training data in professional fields is even scarcer.

[0004] With the rapid development of the Internet, this situation has led to the dilemma of lack of high-quality training data when building large vertical models (high-quality models focusing on specific fields), which has limited the improvement of model performance and in-depth development in specialized applications. Summary of the invention

[0005] The present invention provides a method, device, electronic device and storage medium for constructing a training data set, so as to solve the defects of the prior art in constructing a model training set, such as data source limitations, inconsistent evaluation standards, insufficient professional depth, and difficulty in fully meeting multi-scenario and deep-level professional data requirements.

[0006] The present invention provides a method for constructing a training data set, comprising the following steps: collecting a first data set; Preprocessing the first data set to obtain a second data set, wherein the preprocessing includes converting non-text data in the first data set into text data; Obtaining a comprehensive classification score of data in a target field for each data classification in the second data set, wherein the target field is the field to which the vertical large model to be trained belongs; Based on the comprehensive classification score of each data in the second data set, a target training data set is screened out from the second data set.

[0007] According to a method for constructing a training data set provided by the present invention, the collecting of a first data set includes: For a target data source in any vertical field related to the target field, the following discrimination operations are performed: extracting part of the data samples to construct a vertical field data set, and converting each of the data samples into a text data sample; obtaining a correlation score between each of the text data samples and the target field; based on a comparison result of the correlation score of each of the text data samples with a first preset threshold, determining whether to completely collect the remaining data in the target data source; If confirmed, all the collected data in the target data source are added to the initial data set, and the data source in the next vertical field related to the target field is set as the target data source, and the discrimination operation is re-executed; If uncertain, directly set the next vertical domain data source related to the target domain as the target data source, and re-execute the determination operation; The process is iteratively executed until a preset stop condition is met, and the obtained initial data set is used as the first data set.

[0008] According to the method for constructing a training data set provided by the present invention, obtaining a correlation score between each of the text data samples and the target domain includes: generating a domain-guided text associated with a relevance score of the target domain; Inputting each of the text data samples and the domain-guided text into a pre-trained relevance evaluation model, respectively, to obtain the domain discrimination result and the relevance score for each of the text data samples output by the relevance evaluation model; The domain discrimination result is used to indicate whether the text data sample belongs to the target domain; the relevance score is used to indicate the matching degree between the text data sample and the target domain.

[0009] According to the method for constructing a training data set provided by the present invention, the step of determining whether to completely collect the remaining data in the target data source based on the comparison result of the relevance score of each of the text data samples with a first preset threshold value includes: Obtaining a first number of all text data samples whose domain discrimination results are failed; Obtain a second number of all text data samples whose relevance scores are lower than a second preset threshold; Calculate the proportion of unqualified data according to the first number, the second number, and the total amount of data in the vertical field data set; According to the comparison result of the unqualified data proportion and the first preset threshold, it is determined whether to completely collect the remaining data in the target data source.

[0010] According to the method for constructing a training data set provided by the present invention, for any data in the second data set, obtaining a comprehensive classification score of each data classification in the second data set as data in the target field includes: Inputting any of the data into a text classification model, and obtaining a classification probability of the data output by the text classification model being classified as data within the target domain; Constructing a keyword set related to the target field, and performing a word segmentation operation on any of the data to obtain a vocabulary set, so as to determine a matching score between the vocabulary set and the keyword set; The comprehensive classification score of any one of the data is determined according to a weighted combination of the classification score determined by the classification probability and the matching score.

[0011] According to the method for constructing a training data set provided by the present invention, the step of screening out a target training data set from the second data set based on the comprehensive classification score of each data in the second data set includes: The data whose comprehensive classification score is less than a third preset threshold is eliminated from the second data set to obtain the target training data set.

[0012] According to the method for constructing a training data set provided by the present invention, the text classification model is obtained by pre-training data samples in multiple vertical fields annotated with classification probability labels, and further includes: Obtaining the model performance index of the text classification model after each sample data training, and the comprehensive classification score test value obtained based on each sample data training; Analyze the model performance indicators and comprehensive classification score test values ​​of all the sample data using heat maps and confusion matrices; The weight combination corresponding to each of the vertical fields and the value of the third preset threshold are determined according to the analysis result, and the weight combination is the weight distribution relationship between the classification probability and the matching score.

[0013] According to the method for constructing a training data set provided by the present invention, the preprocessing of the first data set includes: Obtain each HTML format data in the first data set; Locate and strip out the text part of each HTML format data; The text data in each of the body parts is extracted, and redundant formats and useless information in the text data are removed.

[0014] According to the method for constructing a training data set provided by the present invention, the preprocessing of the first data set further includes: Obtain each PDF format data in the first data set; Recognize text blocks in the PDF format data based on optical character recognition technology; According to the positions of the text blocks in the PDF format data, all the text blocks are converted into single-column mode text data after the layout is restored; Remove useless information from the single-column mode text data.

[0015] The present invention also provides a device for constructing a training data set, comprising the following modules: A data source collection module, used for collecting a first data set; A data preprocessing module, used for preprocessing the first data set to obtain a second data set, wherein the preprocessing includes converting non-text data in the first data set into text data; An adaptive domain evaluation module, used to obtain a comprehensive classification score of each data classification in the second data set as data in a target domain, wherein the target domain is the domain to which the vertical large model to be trained belongs; A data set screening module is used to screen out a target training data set from the second data set based on the comprehensive classification score of each data in the second data set.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for constructing a training data set as described above is implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for constructing a training data set as described in any one of the above is implemented.

[0018] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for constructing a training data set as described in any one of the above is implemented.

[0019] The training data set construction method, device, electronic device and storage medium provided by the present invention calculate the comprehensive classification score of each data by introducing an adaptive field evaluation function, and can dynamically evaluate and screen data according to various scenarios and field requirements, thereby having obvious technical improvement effects in broadening data sources, reducing cleaning costs, unifying quality standards and improving the purity of professional field data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flow chart of the method for constructing a training data set provided by the present invention.

[0022] Figure 2 It is a schematic diagram of the process of collecting the first data set provided by the present invention.

[0023] Figure 3 It is one of the structural schematic diagrams of the device for constructing a training data set provided by the present invention.

[0024] Figure 4 This is the second structural schematic diagram of the device for constructing a training data set provided by the present invention.

[0025] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that, in the description of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0028] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] A vertical large model refers to a high-quality network model that focuses on a specific field. Existing technologies for improving the quality of pre-training data for vertical large models mainly focus on the following two points: Optimize data source selection: that is, try to obtain text data from authoritative, professional and screened channels, such as professional databases, scientific research paper libraries and high-credibility clean data sets, fundamentally reducing the proportion of low-quality and irrelevant data mixed in.

[0030] Multiple rounds of refined cleaning and filtering: First, the initial data set is cleaned up through automated tools (such as regular expressions and keyword filtering) to remove advertisements, redundant and obviously irrelevant data. Then, it is filtered multiple times with the help of manual sampling and small quality classifiers to continuously remove suspicious content and low-quality fragments, and finally form a relatively pure and more professional training data set.

[0031] Although the above solution has improved the quality of training data to a certain extent, it is still insufficient when dealing with large vertical models and cannot fully meet the needs of multi-scenario, deep-level specialized data. Its defects are mainly manifested in: (1) Evaluation standards are not unified. Different scenarios and fields have different requirements for the quality of training data, making it difficult to establish universal evaluation standards, which complicates data integration and management.

[0032] (2) Insufficient professional depth. Existing methods focus more on eliminating low-quality or redundant data and lack refined optimization methods for specific professional fields and high-difficulty scenarios, which restricts the professional improvement space of vertical large models.

[0033] (3) Data source limitations: The sources of high-quality training data are limited, the cost of obtaining them is high, and the coverage is not wide enough, making it difficult to meet the needs of large models in multiple scenarios and fields.

[0034] (4) Data cleaning is costly. Multiple rounds of data filtering, manual quality inspection, and classifier optimization require a lot of manpower and time, which is inefficient and not conducive to rapid iteration.

[0035] In view of the above problems, the present invention proposes a method for constructing a training data set for a vertical large model. By introducing an adaptive field evaluation function, data can be dynamically evaluated and screened according to the needs of each scene and field, thereby significantly improving the deficiencies of the existing technology in terms of broadening data sources, reducing cleaning costs, unifying quality standards, and improving the purity of professional field data. Figure 1-Figure 5 The present invention describes a method, device, electronic device and storage medium for constructing a training data set.

[0036] Figure 1 It is a flow chart of the method for constructing a training data set provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps: Step 101: collect a first data set.

[0037] The present invention can obtain potential data sources through various channels, including authoritative databases, industry reports, professional journals, academic paper libraries, standard specification texts, and vertical field data sets provided by industry partners, etc. In order to ensure the breadth and diversity of the collected data, multiple methods such as application programming interface (API) calls, directional crawler capture, and batch import can be used to achieve this.

[0038] For example, for the vertical large model in the medical field, medical-related text data can be collected from the PubMed database, medical journal websites, and the official websites of major medical institutions.

[0039] Step 102: preprocess the first data set to obtain a second data set, wherein the preprocessing includes converting non-text data in the first data set into text data.

[0040] The collected first data set is preprocessed, and the preprocessing steps include but are not limited to data cleaning, format unification, and type conversion. All the preprocessed data constitute a new data set, which is called the second data set.

[0041] Among them, data cleaning is mainly used to remove irrelevant content, such as advertisements, redundant information, and text data that is obviously irrelevant to the vertical model to be trained and obviously erroneous. Format unification refers to unifying the data formats of different sources into a format that is convenient for subsequent processing, such as plain text format or a specific format of JSON format. In particular, in this embodiment, it is necessary to convert the non-text type data (such as pictures, tables, PDF, etc.) in the collected first data set into text type data.

[0042] Step 103, obtaining a comprehensive classification score of each data classification in the second data set as data within the target field.

[0043] Among them, the target domain is the domain to which the vertical large model to be trained belongs. Since different fields have different requirements for data quality, it is difficult to establish a unified standard. In order to evaluate the degree of matching between each data in the second data set and the target domain, the present invention proposes an adaptive domain evaluation function, which selects benchmark source data as a benchmark, for example, selects a well-known data set in the target field as the benchmark source data, and processes the benchmark source data for matching scoring, so that the classification standard is consistent with the benchmark. At the same time, the classification probability obtained by the field classification of each data by the network classifier is combined to calculate the comprehensive classification score of each data, so as to use the comprehensive classification score to achieve a unified evaluation of all data in the second data set.

[0044] Step 104: Filter out a target training data set from the second data set based on the comprehensive classification score of each data in the second data set.

[0045] First, a preset threshold can be set based on experience or historical data and the characteristics of the target field in which the vertical large model to be trained is located. The data in the second data set with a comprehensive classification score greater than or equal to the preset threshold will be retained, and the remaining data with a comprehensive classification score less than the preset threshold will be regarded as data that does not meet the requirements and will be eliminated to construct the target training data set.

[0046] The method for constructing a training data set provided by the present invention introduces an adaptive field evaluation function to calculate the comprehensive classification score of each data, and can dynamically evaluate and screen data according to various scenarios and field requirements, thereby having obvious technical improvement effects in broadening data sources, reducing cleaning costs, unifying quality standards, and improving the purity of professional field data.

[0047] Figure 2 is a schematic diagram of a process for collecting a first data set provided by the present invention, such as Figure 2 As shown, this embodiment will provide a detailed description of the specific implementation method of how to implement the collection of the first data set.

[0048] The premise of constructing training data for a certain vertical model to be trained in the present invention is to use a large model-assisted discrimination strategy to construct an initial data set (called the first data set) with diversity and wide coverage. The main purpose is to quickly filter unsuitable data sources and reduce the cost of data source acquisition. The specific implementation method is as follows: First, data sources can be obtained through a variety of channels, including authoritative databases, industry reports, professional journals, academic paper libraries, standard specification texts, and vertical field data sets provided by industry partners. This module supports multiple methods such as API calls, directional crawlers, and batch imports to ensure the diversity and wide coverage of data sources.

[0049] Among them, the API call method can be used to directly connect with the database of other platforms or other systems to obtain the target data source in any vertical field related to the target field. The advantage of this API call method is that the data source is more accurate and complete, and it can avoid technical obstacles that may be encountered during the crawling process.

[0050] Directed crawling refers to extracting the required data by writing a crawler program for a specific website or data source (generally a website or data source in a vertical field related to the target field). Through directed crawling, data can be flexibly obtained from multiple different data sources, covering a wider range of fields and scenarios. At the same time, directed crawling can also be customized as needed to meet specific data needs.

[0051] Batch import refers to the one-time import of existing data sets or data files. It is mainly suitable for situations where there are a large number of data sources. Batch import can quickly build a large-scale database, which can greatly save time and labor costs.

[0052] Since the amount of unsupervised data collected by the previous data collection method is large, in order to avoid excessive bandwidth and storage consumption, the present invention will first determine multiple vertical fields that are relevant to the target field, and select a data source in any vertical field related to the target field as the target data source, and use the following steps to further screen the target data source.

[0053] First, a small portion of the data (for example, 5%, with a data volume of m) is randomly extracted from the target data source and then the extraction is suspended. The extracted data are used as data samples to build a data set (called a vertical field data set). By analyzing the randomly extracted data samples, it is determined whether to continue extracting data from the target data source, thereby effectively reducing unnecessary resource investment.

[0054] Then, according to the characteristics and requirements of the target data source, select a suitable text extraction tool, such as Trafilatura, and configure the tool parameters according to the specific extraction requirements, so as to use these text extraction tools to efficiently extract and process the text data in the data sample, and automatically identify and remove parts that are irrelevant to the required content, such as advertisements, navigation links, copyright information, etc., to ensure that the extracted text is purer. For example, set the text range to be extracted, filter conditions, etc. to ensure that the extracted text data meets expectations.

[0055] Input the sample data into the configured text extraction tool and perform the extraction operation. The tool will automatically analyze the text structure, extract the required content, and remove irrelevant parts. The rated text data extracted from each data sample is called a text data sample, which can be further verified to ensure that its structure is clear, the content is pure, and it meets the requirements of subsequent quality assessment and analysis.

[0056] Furthermore, for the extracted text data samples, a field guidance text (which can be called a prompt) related to the target field can be designed. For example, if the target field is the medical field, the prompt can be designed as a question such as "Does the text data involve medical terms or medical cases?"

[0057] As a core improvement point, the present invention uses a large model to perform relevance judgment and scoring between each text data sample and the target domain, so as to analyze whether each text data sample belongs to the target domain or a specific application scenario closely related to the target domain based on a preset domain knowledge base and semantic understanding ability through the large model, and obtain the relevance score between each text data sample and the target domain.

[0058] Then, the correlation between each text data sample and the target domain can be determined based on the comparison result of the correlation score and a preset threshold, which can further determine whether to completely collect the remaining data in the target data source based on the correlation scores in all text data samples.

[0059] If it is determined that the data in the target data source is completely collected, the collected data is added to the initial data set, and the data source in the next vertical field related to the target field is set as the new target data source, and the above-mentioned identification operation is performed again.

[0060] If unsure, directly skip the current target data source, set the next vertical domain data source related to the target domain as the new target data source, and re-execute the identification operation.

[0061] All the above processes will be iteratively executed until the preset stop condition is met, and the obtained initial data set will be used as the first data set.

[0062] The preset stop conditions may include but are not limited to the following situations: reaching a predetermined number of iterations, the initial data set reaches a predetermined amount of data, and all data sources in the vertical field related to the target field have been traversed and processed.

[0063] The training data set construction method provided by the present invention can effectively collect data from multiple vertical fields related to the target field, and screen the data according to the correlation score between the data and the target field, and finally obtain a high-quality training data set, providing strong support for the training of large vertical models.

[0064] Based on the content of the above embodiment, as an optional embodiment, obtaining the correlation score between each of the text data samples and the target domain includes: generating a domain-guided text associated with a relevance score of the target domain; Inputting each of the text data samples and the domain-guided text into a pre-trained relevance evaluation model, respectively, to obtain the domain discrimination result and the relevance score for each of the text data samples output by the relevance evaluation model; The domain discrimination result is used to indicate whether the text data sample belongs to the target domain; the relevance score is used to indicate the matching degree between the text data sample and the target domain.

[0065] Combine the following Figure 2 As shown in FIG. 1 , it is described in detail how to score the relevance between a text data sample and a target domain.

[0066] First, according to the characteristics of the target domain, a series of representative questions or instructions are designed and converted into domain-guided text prompts, which are mainly used to provide specific guidance on the target domain or specific application scenarios related to the target domain when the vertical large model assists in judgment, so as to help the model more accurately evaluate the relevance of the text content. For example, in the medical field, the prompt can be designed as "Does the text involve medical terms or medical cases?"

[0067] Optionally, the prompt is taken as part of the text and concatenated with the text data sample to be judged to form a complete input sequence, which is then input into a pre-trained relevance evaluation model. When processing this input sequence, the relevance evaluation model will consider both the prompt and the text data sample to be judged, and make a relevance judgment based on the requirements or instructions in the prompt and give a specific relevance score.

[0068] As another optional embodiment, assuming that the relevance evaluation model is a conditional generation model (such as a conditional variational autoencoder, a conditional generative adversarial network, etc.), Prompt can be used as an additional input condition to guide the discrimination process of the relevance evaluation model.

[0069] The relevance assessment model outputs a relevance score for each text data sample that passes relevance judgment. The score range can be set to 1 to 100. The higher the score, the higher the degree of match with the target field.

[0070] As an optional embodiment, the determining whether to completely collect the remaining data in the target data source based on the comparison result of the relevance score of each of the text data samples with a first preset threshold value includes: Obtaining a first number of all text data samples whose domain discrimination results are failed; Obtain a second number of all text data samples whose relevance scores are lower than a second preset threshold; Calculate the proportion of unqualified data according to the first number, the second number, and the total amount of data in the vertical field data set; According to the comparison result of the unqualified data proportion and the first preset threshold, it is determined whether to completely collect the remaining data in the target data source.

[0071] Specifically, the present invention calculates the proportion of text data samples that meet the field relevance standard in the vertical field data set previously sampled according to the discrimination result of the relevance evaluation model. The specific steps include: Perform relevance screening and count the number of text data samples in the vertical field data set whose field discrimination results are not passed (i.e., do not meet the relevance standard), recorded as the first number .

[0072] Perform relevance score evaluation and count the number of all text data samples in the vertical domain data set whose domain discrimination results are passed but whose relevance scores are lower than the second preset threshold (such as 60), which is recorded as the second number .

[0073] Finally, the proportion of unqualified data is calculated based on the first number, the second number, and the total amount of data in the vertical field data set. The calculation expression is: .

[0074] Set the first preset threshold , if the proportion of unqualified data is greater than or equal to the first preset threshold In the case of a high-quality data source, the crawling of the target data source is terminated, and the crawling is turned to other new data sources with potential high quality, and the above process is repeated.

[0075] On the contrary, if the proportion of unqualified data is less than the first preset threshold This indicates that the quality of the text data in the target data source is high, and you can continue to acquire the full amount of data for the target source data until all data in the entire target source data is collected and then turn to other new data sources that are potential high-quality data sources, or the above-mentioned preset stop conditions are met.

[0076] The method for constructing a training data set provided by the present invention utilizes a large model to assist in domain relevance identification and scoring, performs partial sampling on each data source related to the target domain, and determines whether to continue to complete the full collection of the data source through a small amount of data identification, thereby achieving rapid filtering of unsuitable data sources and reducing the cost of data source acquisition.

[0077] Based on the content of the above embodiment, as an optional embodiment, for any data in the second data set, obtaining the comprehensive classification score of each data classification in the second data set as data in the target field includes: Inputting any of the data into a text classification model, and obtaining a classification probability of the data output by the text classification model being classified as data within the target domain; Constructing a keyword set related to the target field, and performing a word segmentation operation on any of the data to obtain a vocabulary set, so as to determine a matching score between the vocabulary set and the keyword set; The comprehensive classification score of any one of the data is determined according to a weighted combination of the classification score determined by the classification probability and the matching score.

[0078] As an innovative point of the present invention, an adaptive domain evaluation function is proposed . Since different fields have different requirements for data quality and it is difficult to establish a unified standard, the present invention selects benchmark source data (such as well-known data sets in the field) as a benchmark. By processing the benchmark data and using a text classification model (such as the FastText model) to train the classifier, its classification standard is consistent with the benchmark, achieving unified evaluation.

[0079] It should be noted that FastText has higher classification efficiency than deep learning models with large parameters and slow processing (such as BERT). In order to make up for its slightly insufficient language understanding ability, the present invention proposes to combine FastText with a keyword matching mechanism, which not only maintains efficient classification, but also improves the accuracy of screening, ensuring that the data is close to the benchmark scenario. The specific execution process is as follows: Any data in the second data set obtained after data preprocessing can be converted into a vector or embedded representation and then input into the FastText model to obtain the classification probability of the data output by the model being classified as data in the target domain. , this classification probability Convert to a categorical score in the range of 0-100 : .

[0080] Optionally, before using the FastText model, it can be trained through supervised learning or unsupervised learning.

[0081] Furthermore, the specific implementation steps of performing a word segmentation operation on any data to obtain a vocabulary set and determining a matching score between the vocabulary set and the keyword set are: Construct a keyword set K related to the target field }, and obtain the corresponding vocabulary set by performing word segmentation operation on any data to be judged.

[0082] Then, by matching the words in the vocabulary set with the keywords in the keyword set, the matching score between the vocabulary set and the keyword set is calculated according to the number or weight of the matched associated words: ; in, is the number of associated words that appear in the vocabulary set, is the total number of words in the vocabulary.

[0083] Finally, the comprehensive classification score of any data may be determined based on a weighted combination of the classification score determined by the classification probability and the matching score: ; in: is a comprehensive classification score (value range: 0-100), The classification score determined by FastText (value range: 0-100), Score the keyword's match (range: 0-100). and is the weight coefficient, satisfying , which can be dynamically adjusted according to the characteristics of different fields.

[0084] As an optional embodiment, the step of selecting a target training data set from the second data set based on the comprehensive classification score of each data in the second data set includes: The data whose comprehensive classification score is less than a third preset threshold is eliminated from the second data set to obtain the target training data set.

[0085] Specifically, if , then keep the data, otherwise remove the data from the second data set, and finally get the required target training data set. The third preset threshold is different in different fields. The value can be set by yourself.

[0086] Based on the content of the above embodiment, as an optional embodiment, the text classification model is obtained by pre-training with data samples in multiple vertical fields annotated with classification probability labels, and further includes: Obtaining the model performance index of the text classification model after each sample data training, and the comprehensive classification score test value obtained based on each sample data training; Analyze the model performance indicators and comprehensive classification score test values ​​of all the sample data using heat maps and confusion matrices; The weight combination corresponding to each of the vertical fields and the value of the third preset threshold are determined according to the analysis result, and the weight combination is the weight distribution relationship between the classification probability and the matching score.

[0087] As another innovation of the present invention, a dynamic optimization strategy based on feature importance is proposed, which includes dynamically adjusting weights. and , automatically optimize the first preset threshold, the second preset threshold, the third preset threshold, integrate active learning and data enhancement methods, and optimize the vertical large model data through continuous iterative optimization. The specific process is as follows: (1) Model performance monitoring.

[0088] The present invention uses TensorBoard or MLflow to visualize the model performance indicators (such as accuracy, recall rate, F1 score, etc.) monitored in real time. That is, in the process of training the text classification model, the key model performance indicators (such as accuracy, recall rate, F1 score, etc.) can be displayed in a graphical interface in real time, so that users can intuitively understand the performance of the model and make adjustments and optimizations accordingly.

[0089] In addition, during the model training process, various abnormal situations may occur, such as overfitting. In order to detect such situations in a timely manner, an automatic alarm system can be set up. When any model performance indicator (such as the accuracy on the validation set) is lower than a certain threshold after a training round, an alarm will be automatically triggered to remind the user to intervene and make adjustments.

[0090] In addition, during the model training process, it is very important to periodically save the current state of the text classification model (i.e., checkpoints). In this way, even if there is an abnormal interruption during training (such as power failure, hardware failure, etc.), you can resume training from the most recent checkpoint to avoid starting from scratch. At the same time, saving multiple checkpoints can also be used for subsequent performance analysis and model selection.

[0091] (2) Feedback collection and analysis.

[0092] The present invention also performs error analysis based on the collected key model performance indicators and log data of the text classification model to identify common misclassifications and performance bottlenecks.

[0093] Heat maps and confusion matrices are used to display the screening effects of data in different fields, providing guidance for subsequent adjustments and ensuring the targetedness and effectiveness of optimization.

[0094] Among them, the use of heat maps can show the classification effect of data in different fields. The difference in color can intuitively reflect which fields of data are better classified by the model and which fields of data are more difficult to classify. By observing the heat map, you can quickly identify which fields of data are crucial to improving model performance, so that more attention can be paid to them in future data collection and optimization.

[0095] In addition, the confusion matrix can display the accuracy, recall, precision and other indicators of the model in each category, so as to comprehensively evaluate the classification performance of the text classification model. Through the confusion matrix, you can intuitively see which fields of data are easily misclassified, as well as the specific circumstances of the misclassification (such as misclassifying data in a certain field as another field). According to the results of the confusion matrix, it can be clear in which fields the text classification model has performance bottlenecks, thus guiding subsequent optimization work. For example, if the recall rate of a data source in a certain field is low, you can consider increasing the training data of the data source or adjusting the above preset threshold.

[0096] (3) Parameter adjustment and strategy optimization.

[0097] Based on the above feedback and analysis results, the present invention evaluates the text classification model for different evaluation parameters (such as and ). The importance of features (such as SHAP value) is used to determine the sensitivity of different fields. and .

[0098] Furthermore, the threshold can be adjusted by grid search or Bayesian optimization. , balance data quality and quantity.

[0099] You can also introduce active learning to prioritize labeling of data samples that improve model performance; and apply data augmentation technology to generate diversified training samples.

[0100] The training data set construction method provided by the present invention can effectively improve the accuracy and efficiency of data screening through the above measures, ensure the high quality of the data set, and promote the continuous improvement of model performance.

[0101] Based on the content of the above embodiment, as an optional embodiment, the preprocessing of the first data set includes: Obtain each HTML format data in the first data set; Locate and strip out the text part of each HTML format data; The text data in each of the body parts is extracted, and redundant formats and useless information in the text data are removed.

[0102] As an innovation of the present invention, an efficient data processing tool is used in combination with a large model auxiliary strategy to perform fine processing on the first data set obtained in the above embodiment to ensure the structuring of the data and the purity of the content.

[0103] Since the data structure of HTML web pages is complex and contains a lot of irrelevant information, such as advertisements, navigation bars, footnotes, etc., the present invention uses the following method to process them efficiently: (1) HTML parsing technology and natural language processing (NLP) algorithms are used to identify the structure and content of web pages. This can effectively analyze HTML tags and determine which parts contain the main content and which parts are irrelevant information such as advertisements, navigation bars, footnotes, etc.

[0104] (2) By identifying specific HTML tags (such as 、 、 <article>etc.) or a combination of tags, as well as analyzing the attributes of tags (such as class, id, etc.) to accurately locate the body of the web page.

[0105] (3) After identifying the main text, you can also use specialized text extraction tools or algorithms to separate the main text content from HTML tags and remove redundant formatting and useless information, such as removing irrelevant content such as HTML tags, CSS styles, JavaScript codes, and removing redundant spaces, line breaks, special characters, etc. in the text data.

[0106] (4) In order to ensure that the structure of the extracted text data is clear, techniques such as paragraph segmentation and sentence division can be used to organize the text data into a format that is easy to understand and process. At the same time, grammar checking and spelling correction can be performed on the text data to improve its quality and readability.

[0107] (5) The content of the text data can be further purified. During the process, a quality assessment algorithm or model can be used to assess the quality of the extracted text data. These algorithms or models can analyze the language characteristics, content quality and other aspects of the text data, thereby screening out high-quality text data and ensuring that the final output text data is accurate, clear and of high quality.

[0108] The training data set construction method provided by the present invention, when realizing HTML format data processing, can accurately identify the main content area of ​​the web page, eliminate irrelevant information, and purify the extracted text by adopting advanced HTML parsing technology, NLP algorithm, machine learning model and other intelligent processing strategies, thereby ensuring that the final output text data is of high quality and easy to understand and process.

[0109] Based on the content of the above embodiment, as an optional embodiment, the preprocessing of the first data set further includes: Obtain each PDF format data in the first data set; Recognize text blocks in the PDF format data based on optical character recognition technology; According to the positions of the text blocks in the PDF format data, all the text blocks are converted into single-column mode text data after the layout is restored; Remove useless information from the single-column mode text data.

[0110] PDF files often contain complex tables, images or embedded text, and special methods are required to extract the text content. The invention's innovation in PDF data processing includes the following steps: (1) Use the Optical Character Recognition (OCR) engine to efficiently recognize text in PDF and obtain the location information of each text block. Even if the PDF format data contains complex tables, images, or embedded text, the text content can be accurately extracted.

[0111] (2) Perform layout restoration on the recognized text blocks and convert them into single-column mode. This step simplifies the text structure, making subsequent processing and analysis more efficient and convenient.

[0112] Single-column mode refers to converting text content that may originally contain multiple columns and complex layouts into a single, linear text flow. For example, a document in a PDF file originally contains a two-column or three-column layout, which contains various titles, paragraphs, pictures, and tables. When this document is converted to single-column mode, all the content will be rearranged into a continuous, single-column text flow.

[0113] (3) In addition, irregular data commonly found in PDF format data, such as headers, footers, blank lines and other useless information, are structured and removed to keep the logical structure of the single-column mode text data clear.

[0114] In addition, the present invention can also perform some appropriate optimization processing on data in other formats in the first data set, and specifically, scripts can be written according to the data source.

[0115] Figure 3 is one of the structural schematic diagrams of the device for constructing a training data set provided by the present invention, such as Figure 3 The components shown mainly include but are not limited to the following: The data source collection module 31 is mainly used to collect the first data set; A data preprocessing module 32, mainly used to preprocess the first data set to obtain a second data set, wherein the preprocessing includes converting non-text data in the first data set into text data; The adaptive domain evaluation module 33 is mainly used to obtain the comprehensive classification score of each data classification in the second data set as data in the target domain, and the target domain is the domain to which the vertical large model to be trained belongs; The data set screening module 34 is mainly used to screen out a target training data set from the second data set based on the comprehensive classification score of each data in the second data set.

[0116] It should be noted that the training data set construction device provided by the present invention can execute the training data set construction method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0117] The training data set construction device provided by the present invention calculates the comprehensive classification score of each data by introducing an adaptive field evaluation function, and can dynamically evaluate and screen data according to various scenarios and field requirements, thereby having obvious technical improvement effects in broadening data sources, reducing cleaning costs, unifying quality standards and improving the purity of professional field data.

[0118] Figure 4 FIG. 2 is a second structural diagram of the apparatus for constructing a training data set provided by the present invention. Figure 4 As shown, based on the data source collection module, the data preprocessing module and the adaptive domain evaluation and screening module, a data iteration optimization module is also included. Among them, the adaptive domain evaluation and screening module can be regarded as a combination of the adaptive domain evaluation module and the data set screening module.

[0119] Among them, the functions performed by the data source collection module, the data preprocessing module and the adaptive field evaluation and screening module are not described in detail here. In the actual operation process, the corresponding functions recorded in the above embodiments are performed.

[0120] The data iteration optimization module is mainly used to perform the following functions: (1) Model performance monitoring, including obtaining the model performance indicators of the text classification model after each sample data is trained, and the comprehensive classification score test value obtained based on each sample data training.

[0121] (2) feedback collection and analysis, including analysis of model performance indicators and comprehensive classification score test values ​​of all sample data using heat maps and confusion matrices; (3) Parameter adjustment and strategy optimization, including determining the weight combination corresponding to each of the vertical fields and the value of the third preset threshold according to the analysis results, wherein the weight combination is the weight distribution relationship between the classification probability and the matching score.

[0122] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a method for constructing a training data set, the method comprising: collecting a first data set; preprocessing the first data set to obtain a second data set, the preprocessing comprising converting non-text type data in the first data set into text type data; obtaining a comprehensive classification score of each data in the second data set classified as data in a target field, the target field being the field to which the vertical large model to be trained belongs; based on the comprehensive classification score of each data in the second data set, filtering out a target training data set from the second data set.

[0123] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0124] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the method for constructing a training data set provided in the above-mentioned embodiments, the method including: collecting a first data set; preprocessing the first data set to obtain a second data set, the preprocessing including converting non-text type data in the first data set into text type data; obtaining a comprehensive classification score for each data in the second data set classified as data in a target field, the target field being the field to which the vertical large model to be trained belongs; based on the comprehensive classification score of each data in the second data set, filtering out a target training data set from the second data set.

[0125] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the method for constructing a training data set provided in the above-mentioned embodiments, the method comprising: collecting a first data set; preprocessing the first data set to obtain a second data set, the preprocessing comprising converting non-text type data in the first data set into text type data; obtaining a comprehensive classification score for each data in the second data set classified as data in a target field, the target field being the field to which the vertical large model to be trained belongs; based on the comprehensive classification score of each data in the second data set, filtering out a target training data set from the second data set.

[0126] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / article>

Claims

1. A method for constructing a training data set, characterized in that: include: collecting a first data set; Preprocessing the first data set to obtain a second data set, wherein the preprocessing includes converting non-text data in the first data set into text data; Obtaining a comprehensive classification score of data in a target field for each data classification in the second data set, wherein the target field is the field to which the vertical large model to be trained belongs; Based on the comprehensive classification score of each data in the second data set, a target training data set is screened out from the second data set.

2. The method for constructing a training data set according to claim 1, characterized in that: The collecting of the first data set includes: For a target data source in any vertical field related to the target field, the following discrimination operations are performed: extracting part of the data samples to construct a vertical field data set, and converting each of the data samples into a text data sample; obtaining a correlation score between each of the text data samples and the target field; based on a comparison result of the correlation score of each of the text data samples with a first preset threshold, determining whether to completely collect the remaining data in the target data source; If confirmed, all the collected data in the target data source are added to the initial data set, and the data source in the next vertical field related to the target field is set as the target data source, and the discrimination operation is re-executed; If uncertain, directly set the next vertical domain data source related to the target domain as the target data source, and re-execute the determination operation; The process is iteratively executed until a preset stop condition is met, and the obtained initial data set is used as the first data set.

3. The method for constructing a training data set according to claim 2, characterized in that: The obtaining of the correlation score between each of the text data samples and the target domain comprises: generating a domain-guided text associated with a relevance score of the target domain; Inputting each of the text data samples and the domain-guided text into a pre-trained relevance evaluation model, respectively, to obtain the domain discrimination result and the relevance score for each of the text data samples output by the relevance evaluation model; The domain discrimination result is used to indicate whether the text data sample belongs to the target domain; the relevance score is used to indicate the matching degree between the text data sample and the target domain.

4. The method for constructing a training data set according to claim 3, characterized in that: The determining whether to completely collect the remaining data in the target data source based on the comparison result of the relevance score of each of the text data samples with a first preset threshold value includes: Obtaining a first number of all text data samples whose domain discrimination results are failed; Obtain a second number of all text data samples whose relevance scores are lower than a second preset threshold; Calculate the proportion of unqualified data according to the first number, the second number, and the total amount of data in the vertical field data set; According to the comparison result of the unqualified data proportion and the first preset threshold, it is determined whether to completely collect the remaining data in the target data source.

5. The method for constructing a training data set according to claim 1, characterized in that: For any data in the second data set, obtaining a comprehensive classification score of each data classification in the second data set as data in the target field includes: Inputting any of the data into a text classification model, and obtaining a classification probability of the data output by the text classification model being classified as data within the target domain; Constructing a keyword set related to the target field, and performing a word segmentation operation on any of the data to obtain a vocabulary set, so as to determine a matching score between the vocabulary set and the keyword set; The comprehensive classification score of any one of the data is determined according to a weighted combination of the classification score determined by the classification probability and the matching score.

6. The method for constructing a training data set according to claim 5, characterized in that: The step of selecting a target training data set from the second data set based on the comprehensive classification score of each data in the second data set comprises: The data whose comprehensive classification score is less than a third preset threshold is eliminated from the second data set to obtain the target training data set.

7. The method for constructing a training data set according to claim 6, characterized in that: The text classification model is obtained by pre-training with data samples in multiple vertical fields annotated with classification probability labels, and also includes: Obtaining the model performance index of the text classification model after each sample data training, and the comprehensive classification score test value obtained based on each sample data training; Analyze the model performance indicators and comprehensive classification score test values ​​of all the sample data using heat maps and confusion matrices; The weight combination corresponding to each of the vertical fields and the value of the third preset threshold are determined according to the analysis result, and the weight combination is the weight distribution relationship between the classification probability and the matching score.

8. The method for constructing a training data set according to claim 1, characterized in that: The preprocessing of the first data set includes: Obtain each HTML format data in the first data set; Locate and strip out the text part of each HTML format data; The text data in each of the body parts is extracted, and redundant formats and useless information in the text data are removed.

9. The method for constructing a training data set according to claim 1, characterized in that: The preprocessing of the first data set further includes: Obtain each PDF format data in the first data set; Recognize text blocks in the PDF format data based on optical character recognition technology; According to the positions of the text blocks in the PDF format data, all the text blocks are converted into single-column mode text data after the layout is restored; Remove useless information from the single-column mode text data.

10. A device for constructing a training data set, characterized in that: include: A data source collection module, used for collecting a first data set; A data preprocessing module, used for preprocessing the first data set to obtain a second data set, wherein the preprocessing includes converting non-text data in the first data set into text data; An adaptive domain evaluation module, used to obtain a comprehensive classification score of each data classification in the second data set as data in a target domain, wherein the target domain is the domain to which the vertical large model to be trained belongs; A data set screening module is used to screen out a target training data set from the second data set based on the comprehensive classification score of each data in the second data set.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for constructing a training data set as described in any one of claims 1 to 9 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing a training data set as claimed in any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for constructing a training data set as claimed in any one of claims 1 to 9 is implemented.

Images