Standard term library construction method based on artificial intelligence technology

Abstract

The invention discloses a standard term library construction method based on an artificial intelligence technology. The standard term library construction method comprises the following steps: step 1, constructing a system architecture according to business requirements; 2, preparing an sft data set; 3, finely adjusting the model by using the sft data set prepared in the step 2; 4, processing a standard text by using the fine-tuned model, preliminarily extracting standard terms and defining the data; 5, storing the extracted data into a relational database; step 6, carrying out manual examination and correction on data extracted by the model; 7, storing the data subjected to manual examination and correction into a vector database; and 8, development and application: extracting data with relatively high similarity by using an ann algorithm, and displaying the data. According to the standard term library construction method based on the artificial intelligence technology, a mode of combining large model extraction and artificial review is used, and the efficiency and accuracy of term data collection are greatly improved.

Description

Technical Field

[0001] The present invention relates to a terminology library construction method, and more specifically to a standard terminology library construction method based on artificial intelligence technology. Background Art

[0002] The construction of a term base is divided into two parts: term entry and term query. The existing standard term base construction method is to manually process the standard text, and then the business system stores the processed term name and term definition data into the database. When querying terms, the user enters keywords, and the program obtains relevant term information after using keyword fuzzy query in the database, or establishes the data into various index structures developed based on Lucence (such as es, solr, etc.), and performs some simple word segmentation matching to obtain query results.

[0003] However, the current method of building a terminology database has the following defects: 1. Terminology extraction requires a lot of manpower; 2. For a batch of new standard texts, it takes too long from text acquisition to data storage; 3. The accuracy of the query results is not ideal, and it is impossible to find terms with similar meanings but different descriptions. Summary of the invention

[0004] In view of the deficiencies in the prior art, the object of the present invention is to provide a method for constructing a standard terminology library based on artificial intelligence technology that can solve one or more of the above-mentioned deficiencies.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for constructing a standard terminology library based on artificial intelligence technology, characterized in that it comprises the following steps: Step 1: Build system architecture according to business requirements; Step 2: Prepare the sft data set; Step 3: Use the SFT dataset prepared in step 2 to fine-tune the model; Step 4: Use the fine-tuned model to process the standard text and initially extract standard terms and definitions; Step 5, storing the extracted data in a relational database; Step 6: Manually review and correct the data extracted by the model; Step 7: Store the manually reviewed and corrected data into the vector database to complete the construction of the standard terminology database; Step 8: Develop an application and use the ANN algorithm to extract data with high similarity for display.

[0006] As a further improvement of the present invention, the system architecture constructed in step 1 includes: The data collection layer is used to collect standard texts from various sources; The model processing layer is used to allocate computing resources to support model training and operation; The data storage layer is used to store data in a database and ensure data security and fast access; The manual review layer is used for human-computer interaction with external personnel, providing a convenient operation interface to facilitate manual review; The terminal display layer is used to present query results to users.

[0007] As a further improvement of the present invention, the SFT data set prepared in step 2 contains a large amount of standard texts and their corresponding term annotation information, which is used to guide the model to learn how to accurately extract terms.

[0008] As a further improvement of the present invention, the specific method of fine-tuning the model in step three is: using deep learning technology, selecting chatglm4-9b as the pre-trained model, and using the SFT data set prepared in step two to fine-tune the pre-trained model.

[0009] As a further improvement of the present invention, in step seven, before being stored in the vector database, a Chinese word vector embedding model is used to convert the manually reviewed terminology data into a high-dimensional vector form, and then stored in the vector database.

[0010] The beneficial effects of the present invention are as follows: the present invention improves the existing technology based on artificial intelligence technology, uses a fine-tuned large model to preliminarily extract and store standard terms, and then stores the final data in a vector library after manual review. When querying, the ANN algorithm is used to recall term information with a high degree of similarity to the user input content, and presents it in a list form on the terminal.

[0011] The present invention has the following technical effects: using a combination of large model extraction + manual review to greatly improve the efficiency and accuracy of term data collection, using a special data set and annotation method to achieve better SFT effect, greatly increasing the efficiency and accuracy of automatic extraction of the model, greatly reducing manual workload, and using vector query instead of traditional Lucence-based search engines or database direct query in term query, greatly improving the matching range and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of the system architecture constructed by the method for building a standard terminology library based on artificial intelligence technology of the present invention; Figure 2 A schematic diagram of the process of constructing a standard terminology database based on artificial intelligence technology of the present invention; Figure 3 To manually review and correct the interface diagram; Figure 4 A schematic diagram showing the data. DETAILED DESCRIPTION

[0013] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0014] Reference Figure 2 As shown, a method for constructing a standard terminology library based on artificial intelligence technology in this embodiment is mainly used for constructing a standard terminology library in the textile industry, and specifically includes the following steps: Step 1: Establish the overall system architecture The system architecture is built according to business needs. The architecture covers the data collection layer, model processing layer, data storage layer, manual review layer and terminal display layer. In the data collection layer, standard texts from various sources can be efficiently collected; the model processing layer is equipped with powerful computing resources to support the training and operation of the model; the data storage layer uses advanced database technology to ensure data security and fast access; the manual review layer provides a convenient operation interface to facilitate professional review; the terminal display layer presents query results to users in an intuitive and friendly way. The simple network structure diagram of the system can be found in the attached Figure 1 shown.

[0015] Step 2: Prepare the SFT dataset In order to fine-tune the model, we prepared a special SFT (Supervised Fine-tuning) dataset. This dataset contains a large amount of standard text and its corresponding term annotation information, which is used to guide the model to learn how to accurately extract terms. Compared with existing technologies, our dataset is larger and more accurately annotated, which helps to improve the model's term extraction capabilities. The SFT dataset example is shown below: { "instruction": "According to the standard text provided, extract all the Chinese names, English names and definitions of terms mentioned in the terms and definitions section, and generate json output. The specific requirements of json are as follows: - Terms and definitions (data): array, object type - Chinese name of terminology (terminology_ch): string type - English name of terminology (terminology_en): string type - Term definition (definition): string type", "input": "standard text", "output": "```json { "data": [ { "terminology_ch": "aggregate", "terminology_en": "agglomerate", "definition": "A collection of weakly bound particles, an aggregate, or a mixture of both, whose surface area is approximately the same as the sum of the surface areas of the individual particles. Note 1: Aggregates are supported by weak forces, such as van der Waals forces or simple physical entanglements. Note 2: Aggregates are also called secondary particles, while the source particles are called primary particles." }, { "terminology_ch": "aggregate", "terminology_en": "aggregate", "definition": "A new particle composed of particles that are strongly bound or fused together may have a surface area significantly smaller than the sum of the surface areas of its individual particles. Note 1: The forces supporting the aggregate are strong forces, such as covalent bonds or resulting from sintering or complex physical entanglements. Note 2: Aggregates are also called secondary particles, and the source particles are called primary particles." }, { "terminology_ch": "Artificial nanomaterials", "terminology_en": "manufactured nanomaterial", "definition": "Nanomaterials with specific functions or compositions manufactured for commercial purposes." }, { "terminology_ch": "Nanofiber", "terminology_en": "nanofibre", "definition": "A nano-object with two dimensions of similar external dimensions at the nanoscale, and the remaining dimension of one dimension significantly larger than the other two dimensions. Note 1: Nanofibers can be flexible or rigid. Note 2: For two dimensions of similar dimensions, the difference in their external dimensions should be less than 3 times, and the longest external dimension should be 3 larger than the other two dimensions." } ] }```"} Step 3: Fine-tune the model Using deep learning technology, select a suitable pre-trained model (chatglm4-9b is used in this example), and use the SFT dataset prepared in the second step to fine-tune the model. In this example, based on the locally deployed LLaMA-Factory model training platform, the model is fine-tuned with Lora. The training parameters are shown in Table 1: Table 1 Parameter name Parameter Explanation bf16 The calculation precision is bf16 cutoff_len Truncation length, that is, the maximum length of the input sequence after tokenization dataset Dataset name dataset_dir Dataset save path ddp_timeout Distributed Data Parallel Timeout do_train Indicates whether to perform training operation eval_steps Verification interval, that is, the number of update steps between two verifications eval_strategy Verification strategy, here is verification by number of steps finetuning_type Fine-tuning methods flash_attn Self-Attention Mechanism gradient_accumulation_steps Gradient accumulation steps include_num_input_tokens_seen Indicates whether to include the number of input tokens seen learning_rate Initial learning rate logging_steps Print the number of steps in the log lora_alpha LoRA Scaling Factor lora_dropout LoRA weighted random drop probability lora_rank The rank size of the LoRA matrix lora_target LoRA Action Module loraplus_lr_ratio Learning rate multiplier for B matrix in LoRA+ lr_scheduler_type The name of the learning rate regulator max_grad_norm Maximum gradient norm max_samples Maximum number of samples model_name_or_path Pre-trained model save path num_train_epochs Number of training rounds Optim Optimizer Name output_dir Model output path packing Sequence Packing per_device_eval_batch_size Batch size per GPU during validation per_device_train_batch_size Batch size per GPU during training plot_loss Whether to draw the loss curve preprocessing_num_workers Number of preprocessing threads report_to Indicates where to report information, here is none save_steps Save interval, that is, the number of update steps between two breakpoint saves stage Indicates the training stage, here is sft template Indicates the template used, here is glm4 val_size The percentage of the validation set to all samples warmup_steps The number of steps to use for learning rate warmup Through fine-tuning, the model can learn the semantic features and extraction rules of standard terms, providing strong support for subsequent standard text processing. Compared with the un-fine-tuned model, the fine-tuned model has significantly improved in accuracy and recall.

[0016] Step 4: Input the standard text to be processed into the fine-tuned model, which can quickly and accurately extract key information such as terms and definitions. For example, for a 5,000-word silk standard text containing complex professional terms, the model can extract relevant terms and definitions within 30 seconds with an accuracy rate of more than 95%. Compared with traditional manual extraction methods, it greatly improves processing speed and accuracy while reducing labor costs.

[0017] Step 5: Store the processed data into a relational database In order to ensure the stability and maintainability of the data, we store the data extracted by the model in a relational database. This facilitates subsequent data management and manual review operations.

[0018] Step 6: Manual review of the data extracted by the model Professional standard experts are arranged to carefully review and revise the data extracted by the model. Experts evaluate the accuracy, completeness and consistency of terminology based on rich domain knowledge and strict standards. Compared with the existing technology that relies entirely on manual extraction and review, the manual review link of the invention is only an auxiliary means, which can greatly reduce the manual workload and improve the efficiency and quality of data processing. The interface is as follows: Figure 3 shown.

[0019] Step 7: Store the manually reviewed data into the vector database In order to improve query efficiency and accuracy, we use an advanced Chinese word embedding model (bge-large-zh-v1.5 in this example) to convert manually reviewed terminology data into high-dimensional vector form and store it in a vector database. The vector database can support efficient similarity calculation and query operations, allowing us to quickly find terminology information that is highly similar to the user input content. Compared with traditional methods, the method used in this example can more effectively capture the semantic relationship between terms and improve the relevance and comprehensiveness of query results.

[0020] Step 8: Develop an application and use the ANN algorithm to extract data with high similarity for display Develop an application with a user-friendly interface. When a user enters a query keyword, the application uses the ANN algorithm to quickly recall term information with a high degree of similarity to the input content in the vector database, and displays it to the user in a clear and orderly list on the terminal. For example, when a user enters "artificial intelligence technology", the application can return relevant terms such as "machine learning", "deep learning", "natural language processing", etc. within 0.5 seconds, and arrange them from high to low in terms of similarity. Compared with the traditional keyword-based fuzzy query method, this step can provide more accurate and comprehensive query results to meet the diverse needs of users, such as Figure 4 shown.

[0021] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for constructing a standard terminology database based on artificial intelligence technology, characterized in that: The steps include: Step 1: Build system architecture according to business requirements; Step 2: Prepare the sft data set; Step 3: Use the SFT dataset prepared in step 2 to fine-tune the model; Step 4: Use the fine-tuned model to process the standard text and initially extract standard terms and definitions; Step 5, storing the extracted data in a relational database; Step 6: Manually review and correct the data extracted by the model; Step 7: Store the manually reviewed and corrected data into the vector database to complete the construction of the standard terminology database; Step 8: Develop an application and use the ANN algorithm to extract data with high similarity for display.

2. The method for constructing a standard terminology database based on artificial intelligence technology according to claim 1, characterized in that: The system architecture constructed in step 1 includes: The data collection layer is used to collect standard texts from various sources; The model processing layer is used to allocate computing resources to support model training and operation; The data storage layer is used to store data in a database and ensure data security and fast access; The manual review layer is used for human-computer interaction with external personnel, providing a convenient operation interface to facilitate manual review; The terminal display layer is used to present query results to users.

3. The method for constructing a standard terminology database based on artificial intelligence technology according to claim 2 is characterized in that: The SFT dataset prepared in step 2 contains a large amount of standard text and its corresponding term annotation information, which is used to guide the model to learn how to accurately extract terms.

4. The method for constructing a standard terminology database based on artificial intelligence technology according to claim 1, 2 or 3, characterized in that: The specific method of fine-tuning the model in step three is: using deep learning technology, selecting chatglm4-9b as the pre-trained model, and using the SFT data set prepared in step two to fine-tune the pre-trained model.

5. The method for constructing a standard terminology database based on artificial intelligence technology according to claim 1, 2 or 3, characterized in that: In step seven, before being stored in the vector database, the Chinese word vector embedding model is used to convert the manually reviewed terminology data into a high-dimensional vector form, and then stored in the vector database.

Images