A method for constructing an artificial intelligence model training data set

Abstract

The present invention discloses a method for constructing an artificial intelligence model training data set, which specifically includes the following steps: S1, data acquisition: used to automatically collect data from multiple data sources, including but not limited to social media, news websites, public databases and professional field data sources, and select diverse data sources. The present invention relates to the field of artificial intelligence technology. The method for constructing an artificial intelligence model training data set ensures the high quality and accuracy of the data set through comprehensive data preprocessing steps, including cleaning, format conversion, etc., and effectively reduces noise and bias in model training. At the same time, the automated and semi-automated data processing process significantly improves the training efficiency, and the data enhancement step increases the scale and diversity of the data set by generating new data samples, which helps the model to generalize better, thereby improving the accuracy and robustness of the model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for constructing an artificial intelligence model training data set. Background Art

[0002] With the rapid development of artificial intelligence technology, the construction of data sets plays a vital role in model training. However, traditional methods have many shortcomings in data collection, processing and screening, such as uneven data quality, time-consuming and labor-intensive manual labeling, and insufficient data scale. These problems seriously restrict the effectiveness and efficiency of model training. In this regard, we propose a method for constructing an artificial intelligence model training data set to solve the above problems. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a method for constructing an artificial intelligence model training data set, which solves the problems raised in the background technology.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for constructing an artificial intelligence model training data set, specifically comprising the following steps:

[0005] S1. Data collection: used to automatically collect data from multiple data sources, including but not limited to social media, news websites, public databases, and professional data sources;

[0006] S2. Data preprocessing: used to clean, convert and preliminarily process the collected data to improve data quality. Data cleaning includes deduplication, filling missing values ​​and correcting erroneous values. Data format conversion includes converting data into a format suitable for model training and performing normalization and standardization.

[0007] S3, Data labeling: used to label the pre-processed data. The labeling methods include semi-automatic labeling and unsupervised learning labeling. Semi-automatic labeling combines manual and machine intelligence to label the data and provides manual review and correction interfaces. Unsupervised learning labeling uses clustering, classification and other algorithms to preliminarily label the data as a reference for manual labeling.

[0008] S4, Data screening: It is used to screen data based on its quality, field, category and other characteristics to ensure that the data set matches the requirements of model training. Data quality screening is based on indicators such as data accuracy, completeness, and consistency, and field screening is based on data characteristics such as industry classification and application scenarios.

[0009] S5, Data enhancement: used to generate new data samples through data transformation, data synthesis and other methods to increase the scale and diversity of data sets. Data transformation includes synonym replacement, sentence reorganization, etc. Data synthesis includes generating new data samples based on existing data;

[0010] S6. Dataset construction and optimization: It is used to integrate the preprocessed, labeled, screened and enhanced data into the final dataset, and divide it into training set, validation set and test set. At the same time, the dataset is continuously optimized and adjusted according to the feedback results of model training.

[0011] Preferably, the steps S1-S6 include a data acquisition module, a data preprocessing module, a data annotation module, a data screening module, a data enhancement module and a data set construction and optimization module, the output end of the data acquisition module is electrically connected to the input end of the data preprocessing module, the output end of the data preprocessing module is electrically connected to the input end of the data annotation module, the output end of the data annotation module is electrically connected to the input end of the data screening module, the output end of the data screening module is electrically connected to the input end of the data enhancement module, and the output end of the data enhancement module is electrically connected to the input end of the data set construction and optimization module.

[0012] Preferably, the data acquisition module includes a data source selection unit and a data collection unit, the data preprocessing module includes a data cleaning unit and a data format conversion unit, the data annotation module includes a semi-automatic annotation unit and an unsupervised learning annotation unit, the data screening module includes a quality-based data screening unit and a domain-based data screening unit, the data enhancement module includes a data amplification unit and a data balancing unit, and the data set construction and optimization module includes a data set construction unit and a data set optimization unit.

[0013] Preferably, the data acquisition module is used to collect text data from data sources such as social media platforms and news websites, covering comments and articles with different emotional tendencies such as positive, negative and neutral. The data cleaning unit is used to remove irrelevant characters in the text, correct spelling errors, and fill in missing values. The data format conversion unit is used to convert the text data into a format suitable for the NLP model, such as Tokenization and Lowercasing.

[0014] Preferably, the semi-automatic annotation unit is used to combine manual and machine intelligence to annotate the text with sentiment tendency and provide a manual review and correction interface. The unsupervised learning annotation unit uses a sentiment dictionary or a sentiment analysis model to perform preliminary annotation on the text as a reference for manual annotation.

[0015] Preferably, the quality-based data screening unit screens according to indicators such as clarity and relevance of the text to ensure data quality, and the field-based data screening unit is used to ensure that the text data matches the field requirements of the sentiment analysis task, such as e-commerce reviews and movie reviews.

[0016] Preferably, the data amplification unit generates new text samples through synonym replacement and sentence reorganization methods to increase the size and diversity of the data set, and the data balancing unit balances the number of texts with different sentiment tendencies through oversampling or undersampling technology to ensure the balance of the data set.

[0017] Preferably, the dataset construction unit integrates the preprocessed, annotated, screened and enhanced text data into a final dataset, and divides it into a training set, a validation set and a test set. The dataset optimization unit continuously optimizes and adjusts the dataset according to the feedback results of the sentiment analysis model, such as adding new sentiment categories and adjusting the number of texts.

[0018] Beneficial Effects

[0019] The present invention provides a method for constructing an artificial intelligence model training data set. Compared with the prior art, the method has the following beneficial effects:

[0020] (1) Through comprehensive data preprocessing steps, including cleaning and format conversion, the high quality and accuracy of the data set are ensured, effectively reducing the noise and bias in model training. At the same time, the automated and semi-automated data processing process significantly improves the training efficiency. The data enhancement step increases the size and diversity of the data set by generating new data samples, which helps the model to generalize better, thereby improving the accuracy and robustness of the model.

[0021] (2) This method can automatically collect data from multiple data sources, ensuring the diversity and comprehensiveness of the data set. Through the data screening step, appropriate field data can be selected according to actual needs, making the data set more suitable for actual application scenarios. In addition, semi-automatic and unsupervised learning annotation methods provide flexible annotation methods to meet the annotation needs of different fields. The data set construction and optimization steps can be continuously optimized and adjusted according to the feedback results of model training, enhancing the adaptability and flexibility of the data set.

[0022] (3) This method provides new ideas and methods for the development of artificial intelligence technology, which helps promote the development of related scientific research fields. The constructed high-quality data set can be widely used in natural language processing, image recognition and other fields, promoting the application and development of artificial intelligence technology in more fields. At the same time, the proposal and application of this method also helps to promote the rapid development and growth of the artificial intelligence industry, laying a solid foundation for the widespread application of artificial intelligence technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a process flow chart of the present invention;

[0024] Figure 2 is a system block diagram of the present invention;

[0025] Figure 3 It is a schematic diagram of a data acquisition module of the present invention;

[0026] Figure 4 It is a schematic diagram of a data preprocessing module of the present invention;

[0027] Figure 5 This is a schematic diagram of a data annotation module of the present invention;

[0028] Figure 6 It is a schematic diagram of a data screening module of the present invention;

[0029] Figure 7 It is a schematic diagram of a data enhancement module of the present invention;

[0030] Figure 8 Schematic diagram of the data set construction and optimization module of the present invention.

[0031] In the figure: 01, data acquisition module; 02, data preprocessing module; 03, data annotation module; 04, data screening module; 05, data enhancement module; 06, data set construction and optimization module; 011, data source selection unit; 012, data collection unit; 021, data cleaning unit; 022, data format conversion unit; 031, semi-automatic annotation unit; 032, unsupervised learning annotation unit; 041, quality-based data screening unit; 042, domain-based data screening unit; 051, data augmentation unit; 052, data balancing unit; 061, data set construction unit; 062, data set optimization unit. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The present invention provides three technical solutions, including the following embodiments:

[0034] Example 1

[0035] See also Figure 1 , a method for constructing an artificial intelligence model training data set, specifically comprising the following steps:

[0036] S1. Data collection: used to automatically collect data from multiple data sources, including but not limited to social media, news websites, public databases, and professional data sources. Select a variety of data sources, including open source data sets, internal enterprise data, and Internet public data. Use automated scripts or API interfaces to collect data from selected data sources, set a reasonable collection frequency and update strategy, and ensure the timeliness and accuracy of the data.

[0037] S2. Data preprocessing: used to clean, convert the format and preliminarily process the collected data to improve data quality. Data cleaning includes deduplication, filling missing values ​​and correcting erroneous values. Data format conversion includes converting the data into a format suitable for model training and performing normalization and standardization. The collected data is cleaned to remove duplicate data, fill missing values, correct erroneous data, etc. The cleaned data is converted into a unified format and normalized and standardized. According to the needs of model training, appropriate preprocessing operations are performed on the data, such as text segmentation, image cropping, etc.

[0038] S3. Data labeling: used to label the pre-processed data. The labeling methods include semi-automatic labeling and unsupervised learning labeling. Semi-automatic labeling combines artificial and machine intelligence to label the data, and provides a manual review and correction interface. Unsupervised learning labeling uses clustering, classification and other algorithms to preliminarily label the data as a reference for manual labeling. Semi-automatic labeling is used to label the data in combination with artificial and machine intelligence. Unsupervised learning algorithms are used to preliminarily label the data as a reference for manual labeling. An interface for manual review and correction is provided to ensure the accuracy and reliability of the labeling.

[0039] S4. Data screening: It is used to screen data based on its quality, domain, and category to ensure that the data set matches the requirements of model training. Data quality screening is based on data accuracy, completeness, consistency and other indicators. Domain screening is based on data industry classification, application scenarios and other characteristics. Data is screened based on data quality indicators such as accuracy, completeness, consistency, etc. Data is screened based on its domain characteristics to ensure that the data set matches the domain requirements of model training. The screened data is divided into training set, validation set and test set.

[0040] S5, Data enhancement: used to generate new data samples through data transformation, data synthesis and other methods to increase the size and diversity of the data set. Data transformation includes synonym replacement, sentence reorganization, etc. Data synthesis includes generating new data samples based on existing data, using data augmentation technology to increase the size and diversity of the data set, generating new data samples through data transformation, data synthesis and other methods, and using oversampling, undersampling and other technologies to balance the data of various categories in the data set;

[0041] S6. Dataset construction and optimization: It is used to integrate the preprocessed, labeled, screened and enhanced data into the final data set, and divide it into training set, validation set and test set. At the same time, it continuously optimizes and adjusts the data set according to the feedback results of model training, integrates the preprocessed, labeled, screened and enhanced data into the final data set, and provides data set visualization and statistical analysis functions to facilitate users to understand the distribution and characteristics of the data set. According to the feedback results of model training, the data set can be optimized and adjusted in a targeted manner.

[0042] Example 2

[0043] Based on Example 1, see Figure 2-Figure 8 As shown, steps S1-S6 include a data acquisition module 01, a data preprocessing module 02, a data annotation module 03, a data screening module 04, a data enhancement module 05 and a data set construction and optimization module 06. The output end of the data acquisition module 01 is electrically connected to the input end of the data preprocessing module 02, the output end of the data preprocessing module 02 is electrically connected to the input end of the data annotation module 03, the output end of the data annotation module 03 is electrically connected to the input end of the data screening module 04, the output end of the data screening module 04 is electrically connected to the input end of the data enhancement module 05, and the output end of the data enhancement module 05 is electrically connected to the input end of the data set construction and optimization module 06.

[0044] The data acquisition module 01 includes a data source selection unit 011 and a data collection unit 012, the data preprocessing module 02 includes a data cleaning unit 021 and a data format conversion unit 022, the data annotation module 03 includes a semi-automatic annotation unit 031 and an unsupervised learning annotation unit 032, the data screening module 04 includes a quality-based data screening unit 041 and a domain-based data screening unit 042, the data enhancement module 05 includes a data amplification unit 051 and a data balancing unit 052, and the data set construction and optimization module 06 includes a data set construction unit 061 and a data set optimization unit 062.

[0045] Example 3

[0046] Based on Example 2, see Figure 3-Figure 8As shown, the data collection module 01 is used to collect text data from data sources such as social media platforms and news websites, covering comments and articles with different emotional tendencies such as positive, negative and neutral. The data source selection unit 011 is responsible for selecting a variety of data sources to ensure the richness and comprehensiveness of the data set. The data source may include open source data sets, internal enterprise data, and Internet public data. By comprehensively utilizing different data sources, data covering multiple fields and scenarios can be obtained, providing a solid foundation for subsequent model training. The data collection unit 012 is responsible for collecting data from the selected data source. The batch download and extraction of data are realized by using automated scripts or API interfaces. At the same time, by setting a reasonable collection frequency and update strategy, the timeliness and accuracy of the data are ensured. The data cleaning unit 021 is used to remove irrelevant characters in the text, correct spelling errors, and fill in missing values. The data cleaning unit 021 is responsible for cleaning the collected data. The quality and availability of the data are improved by removing duplicate data, filling in missing values, and correcting erroneous data. At the same time, a rule-based and model-based data filtering method is used to further screen out high-quality data. The data format conversion unit 022 is used to convert text data into a format suitable for the NLP model, such as Tokenization and Lowercasing. The data format conversion unit 022 is responsible for converting the cleaned data into a unified format. According to the needs of model training, the data is converted into a form suitable for the input model, such as text, image, audio, etc. At the same time, the data is normalized and standardized to ensure that data from different sources are consistent and comparable when input into the model.

[0047] The semi-automatic annotation unit 031 is used to combine artificial and machine intelligence to annotate the text with sentiment tendency, and provide a manual review and correction interface. The semi-automatic annotation unit 031 adopts a semi-automatic annotation method to annotate the data in combination with artificial and machine intelligence. Through the preset annotation rules and templates, the machine intelligence can automatically complete part of the annotation work, and at the same time provide an interface for manual review and correction to ensure the accuracy and reliability of the annotation. The unsupervised learning annotation unit 032 uses a sentiment dictionary or sentiment analysis model to perform preliminary annotation on the text as a reference for manual annotation. The unsupervised learning annotation unit 032 uses an unsupervised learning algorithm to annotate the data. Through algorithms such as clustering and classification, the characteristics and categories of the data are automatically identified, and preliminary annotation results are provided for the data. These results can be used as a reference for manual annotation to improve the efficiency and accuracy of annotation.

[0048] The quality-based data screening unit 041 screens the data according to indicators such as the clarity and relevance of the text to ensure data quality. The data screening unit 041 screens the data according to the quality indicators of the data. The quality level of the data is evaluated by calculating indicators such as the accuracy, completeness, and consistency of the data. Only data that meets certain quality standards will be retained for subsequent model training. The domain-based data screening unit 042 is used to ensure that the text data matches the domain requirements of the sentiment analysis task, such as e-commerce reviews and movie reviews. The data screening unit 042 screens the data according to the domain characteristics of the data. By identifying the industry classification, application scenarios and other characteristics of the data, ensure that the data set matches the domain requirements of the model training. This helps to improve the performance and accuracy of the model in a specific field.

[0049] The data amplification unit 051 generates new text samples through synonym replacement and sentence reorganization methods to increase the scale and diversity of the data set. The data amplification unit 051 uses data amplification technology to increase the scale and diversity of the data set. New data samples are generated through data transformation, data synthesis and other methods to supplement the original data set. These new samples can enrich the content of the data set and improve the generalization ability of the model. The data balancing unit 052 balances the number of texts with different emotional tendencies through oversampling or undersampling technology to ensure the balance of the data set. The data balancing unit 052 is responsible for balancing the data of each category in the data set. By calculating the proportion of each category of data, the problem of data imbalance is identified. Then, oversampling, undersampling and other technologies are used to adjust the number of data of each category to make the data set more balanced and comprehensive.

[0050] The data set construction unit 061 integrates the preprocessed, annotated, screened and enhanced text data into the final data set, and divides it into a training set, a validation set and a test set. The data set construction unit 061 is responsible for integrating the preprocessed, annotated, screened and enhanced data into the final data set. According to the needs of model training, the data set is divided into a training set, a validation set and a test set. At the same time, the visualization and statistical analysis functions of the data set are provided to facilitate users to understand the distribution and characteristics of the data set. The data set optimization unit 062 continuously optimizes and adjusts the data set according to the feedback results of the sentiment analysis model, such as adding new sentiment categories and adjusting the number of texts. The data set optimization unit 062 is responsible for continuously optimizing the data set. The quality and scale of the data set are continuously improved by regularly updating the data source, introducing new data processing and annotation technologies, adjusting data enhancement strategies, etc. At the same time, according to the feedback results of the model training, the data set is optimized and adjusted in a targeted manner to improve the accuracy and efficiency of the model.

[0051] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0052] The above embodiments of the invention are described in detail, but the contents are only preferred embodiments of the invention and cannot be considered to limit the scope of the invention. All equivalent changes and improvements made within the scope of the invention should still fall within the scope of the invention.

Claims

1. A method for constructing an artificial intelligence model training data set, characterized in that: The specific steps include: S1. Data collection: used to automatically collect data from multiple data sources, including but not limited to social media, news websites, public databases, and professional data sources; S2. Data preprocessing: used to clean, convert and preliminarily process the collected data to improve data quality. Data cleaning includes deduplication, filling missing values ​​and correcting erroneous values. Data format conversion includes converting data into a format suitable for model training and performing normalization and standardization. S3, Data labeling: used to label the pre-processed data. The labeling methods include semi-automatic labeling and unsupervised learning labeling. Semi-automatic labeling combines manual and machine intelligence to label the data and provides manual review and correction interfaces. Unsupervised learning labeling uses clustering, classification and other algorithms to preliminarily label the data as a reference for manual labeling. S4, Data screening: It is used to screen data based on its quality, field, category and other characteristics to ensure that the data set matches the requirements of model training. Data quality screening is based on indicators such as data accuracy, completeness, and consistency, and field screening is based on data characteristics such as industry classification and application scenarios. S5, Data enhancement: used to generate new data samples through data transformation, data synthesis and other methods to increase the size and diversity of the data set. Data transformation includes synonym replacement, sentence reorganization, etc. Data synthesis includes generating new data samples based on existing data; S6. Dataset construction and optimization: It is used to integrate the preprocessed, labeled, screened and enhanced data into the final dataset, and divide it into training set, validation set and test set. At the same time, the dataset is continuously optimized and adjusted according to the feedback results of model training.

2. The method for constructing an artificial intelligence model training data set according to claim 1, characterized in that: The steps S1-S6 include a data acquisition module (01), a data preprocessing module (02), a data annotation module (03), a data screening module (04), a data enhancement module (05) and a data set construction and optimization module (06), wherein the output end of the data acquisition module (01) is electrically connected to the input end of the data preprocessing module (02), the output end of the data preprocessing module (02) is electrically connected to the input end of the data annotation module (03), the output end of the data annotation module (03) is electrically connected to the input end of the data screening module (04), the output end of the data screening module (04) is electrically connected to the input end of the data enhancement module (05), and the output end of the data enhancement module (05) is electrically connected to the input end of the data set construction and optimization module (06).

3. The method for constructing an artificial intelligence model training data set according to claim 2, characterized in that: The data acquisition module (01) comprises a data source selection unit (011) and a data collection unit (012); the data preprocessing module (02) comprises a data cleaning unit (021) and a data format conversion unit (022); the data annotation module (03) comprises a semi-automatic annotation unit (031) and an unsupervised learning annotation unit (032); the data screening module (04) comprises a quality-based data screening unit (041) and a domain-based data screening unit (042); the data enhancement module (05) comprises a data amplification unit (051) and a data balancing unit (052); and the data set construction and optimization module (06) comprises a data set construction unit (061) and a data set optimization unit (062).

4. The method for constructing an artificial intelligence model training data set according to claim 3, characterized in that: The data collection module (01) is used to collect text data from data sources such as social media platforms and news websites, covering comments and articles with different emotional tendencies such as positive, negative and neutral. The data cleaning unit (021) is used to remove irrelevant characters in the text, correct spelling errors, and fill in missing values. The data format conversion unit (022) is used to convert the text data into a format suitable for the NLP model, such as Tokenization and Lowercasing.

5. The method for constructing an artificial intelligence model training data set according to claim 3, characterized in that: The semi-automatic annotation unit (031) is used to combine manual and machine intelligence to annotate the text with sentiment tendency and provide a manual review and correction interface. The unsupervised learning annotation unit (032) uses a sentiment dictionary or a sentiment analysis model to perform preliminary annotation on the text as a reference for manual annotation.

6. The method for constructing an artificial intelligence model training data set according to claim 3, characterized in that: The quality-based data screening unit (041) screens the text according to indicators such as clarity and relevance to ensure data quality, and the domain-based data screening unit (042) is used to ensure that the text data matches the domain requirements of the sentiment analysis task, such as e-commerce reviews and movie reviews.

7. The method for constructing an artificial intelligence model training data set according to claim 3, characterized in that: The data amplification unit (051) generates new text samples through synonym replacement and sentence reorganization methods to increase the scale and diversity of the data set. The data balancing unit (052) balances the number of texts with different sentiment tendencies through oversampling or undersampling technology to ensure the balance of the data set.

8. The method for constructing an artificial intelligence model training data set according to claim 3, characterized in that: The data set construction unit (061) integrates the preprocessed, annotated, screened and enhanced text data into a final data set, and divides it into a training set, a validation set and a test set. The data set optimization unit (062) continuously optimizes and adjusts the data set according to the feedback results of the sentiment analysis model, such as adding new sentiment categories and adjusting the number of texts.

Images