Meteorological corpus construction method, system and related device

Abstract

The invention discloses a meteorological corpus construction method and system and a related device, and belongs to the technical field of meteorological data processing. The method comprises the following steps: collecting an original corpus file in the meteorological field, and converting the original corpus file into uniform Markdown format data; cleaning the data in the Markdown format, and organizing the data in the Markdown format into a uniformly structured meteorological corpus; and combining the uniformly structured meteorological corpus with an external knowledge base to construct various data sets required by the meteorological service large model, wherein the various data sets comprise a data set for pre-training, a data set for fine tuning, and a data set and an evaluation benchmark data set required for retrieval enhancement generation. According to the method, the construction efficiency is remarkably improved, the labor cost is reduced, and meanwhile, the quality and consistency of the meteorological corpus are ensured.

Description

Technical Field

[0001] The invention proposes a meteorological corpus construction method, system and related devices, belonging to the technical field of meteorological data processing. Background Art

[0002] The rise of large language models such as ChatGPT has greatly increased people's attention to the construction and application of large-scale, high-quality corpora. The commonly used corpora at home and abroad are usually oriented to general fields, and the knowledge processing of professional fields is relatively rough, which makes it difficult for general large language models to meet the needs of users in professional fields for precise knowledge services. In response to this challenge, specific fields such as agriculture, medicine, finance, and law are actively building and improving their own professional corpora to develop large language models for specific industries. Although the meteorological field has rich professional corpus resources, a special meteorological professional meteorological corpus has not yet been formed. Building a professional corpus for the meteorological field is an important basis for building a large model of meteorological services. In view of the current lack of professional corpora in the meteorological field, the development of large models of meteorological services has been seriously restricted. Therefore, it is urgent to start building a comprehensive and accurate meteorological corpus, which will not only effectively make up for the existing difficulties, but also lay a solid data foundation for the establishment of large models in the meteorological professional field.

[0003] Given the unique comprehensive nature of the meteorological field, which covers a wide range of disciplines such as atmospheric science, ocean science, agricultural meteorology, and information science, the text data in this field is inevitably filled with a large number of complex and professional meteorological terms, such as "Southwest Cold Nest", "Tropical Storm", "Northeast Cold Nest", "Cold Wave and Cold Current", etc. These terms are not only huge in number, but also profound in meaning. The general meteorological corpus construction method cannot meet the specific needs of this field. In addition, in order to support tasks such as pre-training, fine-tuning, and retrieval enhancement generation of large models for meteorological services, it is also required to deliberately build large-scale, diverse, and high-quality datasets for the corresponding tasks. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention proposes a meteorological corpus construction method, system and related devices, which significantly improves the construction efficiency, reduces labor costs, and also ensures the quality and consistency of the meteorological corpus.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: In a first aspect, the present invention provides a method for constructing a meteorological corpus for a large meteorological service model, comprising: Collect original corpus files in the meteorological field and convert them into unified markdown format data; Cleaning the markdown format data and organizing it into a unified structured meteorological corpus; The unified structured meteorological corpus is combined with an external knowledge base to construct various data sets required for a large meteorological service model. The various data sets include pre-trained data sets, fine-tuned data sets, data sets required for retrieval enhancement generation, and evaluation benchmark data sets.

[0006] As a further improvement of the present invention, the original corpus files in the meteorological field include various weather decision-making service files, business files in the meteorological field, training materials, various papers and teaching materials, and various popular science knowledge. The format types of the original corpus files include doc, docx, pdf, text, xlsx, ofd, caj, html and picture text data.

[0007] As a further improvement of the present invention, the conversion of original corpus files of various formats into files in markdown format with a preset meta information template specifically includes: According to the format type of the original corpus file, a conversion program of the corresponding format is written, and the original corpus files of various formats are converted into markdown format by the conversion program. Specifically, the doc file is first converted into a docx file and then converted into markdown format using pandoc; ofd and caj files are first automatically converted into pdf format by the program, and then the pdf content is converted into plain text format using Pdfminer; for pdf files or image type text files that failed to be converted by Pdfminer, the text content is recognized and converted into plain text format according to the OCR model; Then, a distributed computing framework is used to establish a parallel data conversion pipeline to process large-scale corpus data.

[0008] As a further improvement of the present invention, the markdown format data is cleaned and organized into a unified structured meteorological corpus, including: First, apply regular expressions to filter out images, tables, headers, footers, page numbers, URLs, references, and identify and anonymize personally identifiable information; and remove extra spaces, line breaks, and other non-text characters, and replace or eliminate special characters, emoticons, and garbled characters; Then, a multi-class hashing-based method is used to eliminate duplicate data through multi-angle comparison; Then, the fuzzy time words in the weather bulletin and / or special report are replaced with specific time words; Finally, the specific time words are organized into a unified structured meteorological corpus form.

[0009] As a further improvement of the present invention, the process of constructing the pre-trained data set includes: The pre-built fine-cleaning module and data analysis module are used to perform quality filtering on the data in the form of a unified structured meteorological corpus; after quality filtering, the data set required for pre-training is obtained; The data analysis module analyzes the statistical characteristics of the data and presents the data distribution using a visualization method; the fine cleaning module identifies and removes data that is not cleaned by the regular expression, including a heuristic rule-based method and a classifier-based method; As a further improvement of the present invention, the heuristic rule-based method is used to identify and eliminate low-quality data, and the heuristic rules include: 1) Eliminate non-Chinese texts; 2) Downsampling of less useful content; less useful content includes weather bulletins, special reports, and express reports that tend to be more colloquial; The classifier-based method is used to handle situations that are beyond the scope of the heuristic rule-based method, including: using a perplexity scorer to calculate the perplexity score, and discarding data with perplexity scores higher than the average perplexity score.

[0010] As a further improvement of the present invention, the process of constructing the data set required for the retrieval enhancement generation includes: Introducing an external vector knowledge base and a meteorological knowledge graph to perform retrieval enhancement generation to obtain a data set required for the retrieval enhancement generation; Among them, the method for constructing a meteorological knowledge graph includes: based on a large language model, extracting and summarizing nodes, edges and covariates to construct a knowledge graph; reviewing and filling in the gaps in the constructed knowledge graph to obtain the final meteorological knowledge graph.

[0011] As a further improvement of the present invention, the process of constructing the evaluation benchmark data set includes: The evaluation benchmark data set was obtained by adapting the questions from previous forecaster training examinations and providing meteorological professional questions and answers through review and screening.

[0012] In a second aspect, the present invention provides a meteorological corpus construction system for a large meteorological service model, comprising: The format conversion module is used to collect the original corpus files in the meteorological field and convert the original corpus files into unified markdown format data; A data cleaning and structuring module is used to clean the markdown format data and organize it into a unified structured meteorological corpus; The data set construction module is used to combine the unified structured meteorological corpus with an external knowledge base to construct various data sets required for the meteorological service large model. The various data sets include data sets for pre-training, data sets for fine-tuning, data sets required for retrieval enhancement generation, and evaluation benchmark data sets.

[0013] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for constructing a meteorological corpus for a large meteorological service model when executing the computer program.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for constructing a meteorological corpus for a large meteorological service model.

[0015] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions instruct a computer to execute the method for constructing a meteorological corpus for a large meteorological service model.

[0016] The beneficial effects of the present invention compared with the prior art are as follows: The present invention adopts an automated meteorological corpus construction process, including steps such as data collection, format conversion, cleaning and structuring, and combines with an external knowledge base to construct a meteorological corpus for specific needs in the field of meteorological services, which significantly improves construction efficiency, reduces labor costs, and also ensures the quality and consistency of the meteorological corpus. By constructing a large-scale meteorological corpus in the field of Chinese meteorological services, a wide range of meteorological knowledge and information can be covered. This provides rich data resources for the training and application of meteorological service large models, and helps to improve the accuracy and generalization ability of meteorological service large models. The meteorological corpus adopts a structured organization method, which is convenient for researchers to retrieve and apply specific knowledge, and helps to promote research and development in the field of meteorological services, and improve the level and quality of meteorological services. The constructed data set includes a pre-trained data set, a fine-tuned data set, a data set required for retrieval enhancement generation, and an evaluation benchmark data set, which can support a variety of application scenarios and needs. It also helps the meteorological service large model to perform well in different tasks and scenarios, providing a more comprehensive and accurate solution for the field of meteorological services. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flow chart of the meteorological corpus construction method provided by the present invention; Figure 2 The overall architecture diagram of the meteorological corpus provided by the present invention; Figure 3 This is a schematic diagram of the unified format of data; Figure 4 A flowchart of the data cleaning process for meteorological corpus pre-training; Figure 5 A meteorological corpus construction device for a large meteorological service model provided by the present invention; Figure 6 A schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0020] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0021] Explanation of relevant terms Markdown format is a lightweight markup language that allows people to write documents in a plain text format that is easy to read and write, and then convert it into valid HTML (or other formats). By using simple text formats to represent layout elements such as titles, lists, links, images, code blocks, etc., document writing and layout become simpler and more intuitive.

[0022] Fine-tuning a dataset refers to the process of further training a pre-trained model using a new dataset. This process is usually used to adapt the model to new tasks or fields and improve the performance of the model on specific tasks.

[0023] Retrieval-Augmented Generation (RAG) is an innovative method in the field of natural language processing (NLP). It combines the advantages of retrieval-based and generation-based models to improve the quality of generated text.

[0024] Pandoc is a free command line software based on the GPL open source protocol, which is used to convert documents in different formats. It supports multiple input and output formats, such as Markdown, HTML, LaTeX, etc., and can easily convert documents from one format to another.

[0025] Python package is a way to organize modules, which is used to group related modules together. The main purpose of Python package is to provide a hierarchical organizational structure to better organize and manage Python code.

[0026] Pdfminer is a tool for extracting information from PDF documents. It can parse PDF files and provide different levels of data extraction functions. Pdfminer can parse text, comments, form data, etc. The Optical Character Recognition (OCR) model is a technology used to extract text from images. OCR models are widely used in computer vision and natural language processing, such as converting scanned documents into editable text files, automatically reading license plate numbers, and processing handwritten text.

[0027] Heuristic rules are rules based on experience and intuition, used to solve complex problems or make decisions. Unlike traditional precise rules or algorithms, heuristic rules rely on the experience and intuition of experts to formulate rules. Heuristic rules are usually used for problems that are difficult to describe with precise mathematical models, or when the scale of the problem is too large to be solved with traditional methods. Heuristic rules are widely used, such as artificial intelligence, machine learning, data mining and other fields, where heuristic rules are often used to optimize algorithms or improve model performance.

[0028] GraphRAG is a method for improving how computers understand and use information, and stands for Graph-based Retrieval Augmented Generation.

[0029] Retrieval-Augmented Generation (RAG) is a natural language processing (NLP) technology that combines the advantages of information retrieval and language generation. It first retrieves relevant information from an external knowledge source (such as a document database, knowledge graph, etc.), and then uses the retrieved information to help generate high-quality natural language text, such as answering questions and generating document summaries.

[0030] like Figure 1As shown, the first object of the present invention is to provide a method for constructing a meteorological corpus for a large meteorological service model, comprising: S1, collect original corpus files in the meteorological field and convert them into unified markdown format data; S2, cleaning the markdown format data and organizing it into a unified structured meteorological corpus; S3, combining the unified structured meteorological corpus with an external knowledge base to construct various data sets required for the meteorological service large model, including data sets for pre-training, data sets for fine-tuning, data sets required for retrieval enhancement generation, and evaluation benchmark data sets.

[0031] The above scheme provides a method for building a meteorological corpus for a large meteorological service model. The principle is to build a large-scale, structured Chinese meteorological service meteorological corpus based on the needs of the meteorological service field through a systematic method to support the development and application of a large language model for Chinese meteorological service decision-making. The data collection and conversion process is as follows: First, collect original corpus files in the meteorological field, including various weather decision-making service files, business files in the meteorological field, training materials, various papers and teaching materials, various popular science knowledge, etc.

[0032] Then, we write targeted programs to automatically convert these original meteorological corpus files into unified markdown format data. The Markdown format is simple and readable, which facilitates subsequent data cleaning and structured organization. During the data cleaning and structuring process: the converted markdown format data is fully cleaned, quality filtered and deduplicated.

[0033] Then, according to the characteristics and needs of the meteorological service field, the data is organized into a unified structured meteorological corpus. This structured organization is helpful for subsequent data processing and analysis, and also facilitates researchers to retrieve and apply specific knowledge.

[0034] During the dataset construction process, various datasets required for the meteorological service large model are extracted and constructed from the structured meteorological corpus by combining external knowledge bases (such as decision service knowledge base, real-time updated meteorological news and weather condition library, etc.).

[0035] These datasets include pre-trained datasets for incremental model pre-training, datasets for fine-tuning for specific tasks, datasets that support retrieval enhancement generation techniques, and benchmark datasets for evaluating model performance. The construction of these datasets provides data support for the meteorological service large model, ensuring that the model has professional performance in the field of meteorological services.

[0036] The method of the present invention can serve the large language model of Chinese meteorological service decision-making, and construct a meteorological corpus for specific needs in the field of meteorological services. An automated meteorological corpus construction process is adopted, including steps such as data collection, conversion, cleaning and structuring. A large-scale meteorological corpus in the field of Chinese meteorological services is constructed, covering a wide range of meteorological knowledge and information. A structured organization method is adopted to facilitate researchers to retrieve and apply specific knowledge. The constructed data sets include pre-trained data sets, fine-tuned data sets, data sets required for retrieval enhancement generation, and evaluation benchmark data sets, which can support a variety of application scenarios and needs.

[0037] This helps the model perform well in different tasks and scenarios, and provides more comprehensive and accurate solutions for the meteorological service field. Finally, a large-scale meteorological corpus in the field of Chinese meteorological services was constructed, which is specifically used for large language models for Chinese meteorological service decisions and is suitable for services such as weather forecast decisions. The meteorological corpus construction process is automated, which significantly improves construction efficiency and reduces labor costs. The meteorological corpus provides important data support for the research and development of large language models in the field of Chinese meteorological services, and also facilitates subsequent researchers to retrieve and apply various professional knowledge in the meteorological corpus in this field.

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] See also Figure 2 The present invention constructs a meteorological corpus in the Chinese meteorological field, facing the meteorological service large model, and the construction method includes the following steps: S1: Collect the original meteorological corpus and convert it into a unified markdown format data; Step S1 specifically includes the following steps: relying on various units of the China Meteorological Administration to collect various original corpus files, the original corpus files specifically include various weather decision-making service files, business files in the meteorological field, training materials, various papers and teaching materials, various popular science knowledge and other original corpus files, these original corpus files have different format types, including doc, docx, pdf, text, xlsx, ofd, caj, html and picture text data and other format types, it is necessary to convert the above various format types of original corpus files into the following formats: Figure 3 The markdown format file containing various meta information of the original corpus file is shown. The main content of the original corpus file is stored in Figure 3 'fulltext' shown.

[0040] Furthermore, in order to unify the format types, a specific format conversion program is written according to the format type of the original corpus file. The specific doc file is first converted into a docx file and then converted into markdown format using pandoc; the ofd and caj files are first automatically converted into pdf format by the program, and then the Pdfminer of the Python package is used to convert the pdf content into plain text format; for pdf files or image type text files that fail to be converted by Pdfminer, the text content is recognized and converted into plain text format based on the locally deployed OCR model. In addition, an efficient parallel data conversion pipeline is established based on distributed computing frameworks such as Ray (a distributed computing framework), which can quickly process large-scale corpus data. Currently, the number of processed sub-files has reached 1000k+.

[0041] S2: The original corpus is cleaned and organized into a unified structured meteorological corpus for subsequent meteorological corpus retrieval and application; Step S2 is data cleaning to eliminate low-quality, redundant, irrelevant, or even potentially harmful data. It specifically includes the following steps: Apply regular expressions to filter out images, tables, headers, footers, page numbers, URLs, references, and identify and anonymize personally identifiable information (PII) (telephone numbers, email addresses, IP addresses). In addition, any extra spaces, line breaks, and other non-text characters are deleted, and special characters, emoticons, and garbled characters are also replaced or eliminated.

[0042] In order to reduce the risk of overfitting during pre-training and improve its generalization ability, this embodiment also uses a multi-class hash-based method, such as minHash, simHash, MD5 hash value and other multi-algorithm multi-angle comparison to eliminate duplicate data. Since the original corpus comes from meteorological professionals, the compilation results have been reviewed and follow moral and ethical principles.

[0043] It is worth noting that the only words referring to time in some weather bulletins and / or special reports are "today", "yesterday" and "tomorrow", which are not covered by the above cleaning process. To address this problem, when converting the format in step S1, the time content in the data is directly extracted, such as "January 2, 2024", and the time content is replaced with the above words, such as "today" -> "January 2, 2024", "yesterday" -> "January 1, 2024", and so on.

[0044] In addition, some of the business products in the collected original corpus contain image information, such as weather bulletins, special reports, express reports, etc., which contain descriptions of various forecast maps. These meteorological image information is of great help in improving the current multimodal large model's geographic information processing capabilities and meteorological knowledge comprehension capabilities. Based on this type of data, a meteorological multimodal data set is constructed, specifically referring to image and text modal data. By designing an automatic detection algorithm for image data and a regular algorithm for matching images and corresponding texts, image and text pairs are extracted. For example, according to the image index, the paragraph corresponding to the image index (the description of the image) is queried to form an image-text pair. After the multimodal data is constructed, the above text data cleaning method is used to filter and deduplicate the multimodal data, and manual quality inspection and correction are performed based on the open source intelligent data annotation platform.

[0045] S3 builds application datasets such as pre-training and fine-tuning for large models of meteorological services.

[0046] Step S3 includes various data sets required for building a large meteorological service model. Specific data sets include data sets required for pre-training, fine-tuning, and retrieval enhanced generation (RAG) and evaluation benchmark data sets for evaluating the performance of building professional models.

[0047] The capabilities of the basic model rely heavily on large-scale, diverse, and high-quality pre-training data. For the construction of pre-training data, after the data cleaning process in step S2, the quality of the data is guaranteed to a certain extent, but it may not be suitable for the pre-training task of the model. To address the above issues, a fine cleaning module and a data analysis module are constructed for further quality filtering.

[0048] The data analysis module analyzes the statistical characteristics of the data and uses visualization methods to present the data distribution (text length, perplexity, word count, paragraph length, language, etc.) in a user-friendly manner. The fine cleaning module is designed to specifically identify and remove low-quality data that has not been cleaned in step S2, including heuristic rule-based methods and classifier-based methods.

[0049] Based on the heuristic rule method, the designed rules are used to specifically identify and eliminate low-quality data. Combined with the cleaning in step S2 and the high-quality characteristics of the original collected corpus, the designed rules include: 1) Since the goal is to build a Chinese meteorological corpus, non-Chinese texts are removed. In this embodiment, non-Chinese texts are mainly English.

[0050] 2) Downsample less useful content (mainly weather bulletins and / or special reports that tend to be more colloquial) to ensure information density.

[0051] Specifically, "downsampling" means filtering and compressing information, removing redundant or unimportant parts, and highlighting the main content and core ideas. Downsampling of less useful content means that information is being streamlined and optimized to ensure that readers or listeners can obtain the most important information faster while reducing unnecessary reading or listening burden.

[0052] The classifier-based approach handles situations that are beyond the scope of the heuristic rule-based approach, including: the perplexity scorer discards documents whose perplexity scores are far above the average level. The specific criteria can be set by setting a threshold for judgment, such as setting the threshold to a multiple of the average level.

[0053] In addition, external knowledge bases, including vector knowledge base and meteorological knowledge graph, are constructed for retrieval augmentation generation (RAG) technology to reduce model hallucinations and supplement real-time information. The construction method of the meteorological knowledge graph is based on the construction method of the knowledge graph in GraphRAG. The specific process is as follows: First, clean text is obtained from the source document using the cleaning method of step S2. Second, GLM4-130B is used to extract and summarize nodes (such as entities), edges (such as relationships), and covariates (such as statements) to construct a knowledge graph. Finally, the constructed knowledge graph is manually reviewed and supplemented.

[0054] As an example, the evaluation benchmark data set of the present invention is to evaluate the professionalism of the constructed meteorological service model. The evaluation benchmark data set is adapted from meteorological professional examination questions such as pre-job training and professional training of forecasters in previous years, as well as meteorological professional questions and answers given by several meteorological experts, and then reviewed and screened by multiple meteorological experts, covering 10 directions such as meteorological principles, general atmospheric science, dynamic meteorology, radar meteorology, landscape and dynamics, and 2.7k+ questions. The review and screening of multiple meteorological experts can obtain a meteorological expert model through training, and the meteorological expert model is used for review and screening, reducing the workload of manual processing.

[0055] like Figure 5 As shown, the third object of the present invention is to provide a meteorological corpus construction system for a large meteorological service model, comprising: The data collection and conversion module is used to collect original corpus files in the meteorological field and convert them into unified markdown format data; A data cleaning and structuring module is used to clean the markdown format data and organize it into a unified structured meteorological corpus; The data set construction module is used to combine the unified structured meteorological corpus with an external knowledge base to construct various data sets required for the meteorological service large model. The various data sets include data sets for pre-training, data sets for fine-tuning, data sets required for retrieval enhancement generation, and evaluation benchmark data sets.

[0056] The meteorological corpus construction system for a meteorological service large model of the present invention is based on a meteorological corpus construction method for a meteorological service large model.

[0057] like Figure 6 As shown, the third purpose of the embodiment of the present invention is to provide an electronic device, including a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor, wherein the processor implements the above-mentioned method for constructing a meteorological corpus for a large meteorological service model when executing the computer program. It also includes a communication interface 703 and a bus 704.

[0058] A fourth objective of an embodiment of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for constructing a meteorological corpus for a large meteorological service model is implemented.

[0059] A fifth objective of an embodiment of the present invention is to provide a computer program product, which includes computer instructions, and the computer instructions instruct a computer to execute the above-mentioned meteorological corpus construction method for a large meteorological service model.

[0060] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0062] The present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, readable storage media, optical storage, etc.) containing computer-usable program codes.

[0063] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for constructing a meteorological corpus for a large meteorological service model, characterized by: The following steps are involved: Collect original corpus files in the meteorological field, and convert original corpus files of various formats into markdown files with preset metadata templates; Cleaning the markdown format data and organizing it into a unified structured meteorological corpus; The step of cleaning the markdown format data and organizing it into a uniformly structured meteorological corpus includes: First, apply regular expressions to filter out images, tables, headers, footers, page numbers, URLs, references, and identify and anonymize personally identifiable information; and remove extra spaces, line breaks, and other non-text characters, and replace or eliminate special characters, emoticons, and garbled characters; Then, a multi-class hashing-based method is used to eliminate duplicate data through multi-angle comparison; Then, the fuzzy time words in the weather bulletin and / or special report are replaced with specific time words; Finally, the specific time words are organized into a unified structured meteorological corpus; The unified structured meteorological corpus is combined with an external knowledge base to construct various data sets required for a large meteorological service model. The various data sets include pre-trained data sets, fine-tuned data sets, data sets required for retrieval enhancement generation, and evaluation benchmark data sets.

2. The method for constructing a meteorological corpus for a large meteorological service model according to claim 1, characterized in that: The original corpus files in the meteorological field include various weather decision-making service files, business files in the meteorological field, training materials, various papers and teaching materials, and various popular science knowledge. The format types of the original corpus files include doc, docx, pdf, text, xlsx, ofd, caj, html and picture text data.

3. The method for constructing a meteorological corpus for a large meteorological service model according to claim 1, characterized in that: The step of converting the original corpus files of various formats into files in the markdown format of the preset meta-information template specifically includes: According to the format type of the original corpus file, a conversion program of the corresponding format is written, and the original corpus files of various formats are converted into markdown format by the conversion program. Specifically, the doc file is first converted into a docx file and then converted into markdown format using pandoc; ofd and caj files are first automatically converted into pdf format by the program, and then the pdf content is converted into plain text format using Pdfminer; for pdf files or image type text files that failed to be converted by Pdfminer, the text content is recognized and converted into plain text format according to the OCR model; Then, a distributed computing framework is used to establish a parallel data conversion pipeline to process large-scale corpus data.

4. The method for constructing a meteorological corpus for a large meteorological service model according to claim 1, characterized in that: The construction process of the pre-trained data set includes: The pre-built fine-cleaning module and data analysis module are used to perform quality filtering on the data in the form of a unified structured meteorological corpus; after quality filtering, the data set required for pre-training is obtained; The data analysis module analyzes the statistical characteristics of the data and presents the data distribution using a visualization method; the fine cleaning module identifies and removes data that is not cleaned by the regular expression, including a heuristic rule-based method and a classifier-based method; The heuristic rule-based method is used to identify and eliminate low-quality data. The heuristic rules include: 1) Eliminate non-Chinese texts; 2) Downsampling of less useful content; less useful content includes weather bulletins, special reports, and express reports that tend to be more colloquial; The classifier-based method is used to handle situations that are beyond the scope of the heuristic rule-based method, including: using a perplexity scorer to calculate the perplexity score, and discarding data with perplexity scores higher than the average perplexity score.

5. The method for constructing a meteorological corpus for a large meteorological service model according to claim 1, characterized in that: The process of constructing the data set required for the retrieval enhancement generation includes: Introducing an external vector knowledge base and a meteorological knowledge graph to perform retrieval enhancement generation to obtain a data set required for the retrieval enhancement generation; Among them, the method for constructing a meteorological knowledge graph includes: based on a large language model, extracting and summarizing nodes, edges and covariates to construct a knowledge graph; reviewing and filling in the gaps in the constructed knowledge graph to obtain the final meteorological knowledge graph.

6. The method for constructing a meteorological corpus for a large meteorological service model according to claim 1, characterized in that: The construction process of the evaluation benchmark dataset includes: The evaluation benchmark data set was obtained by adapting the questions from previous forecaster training examinations and providing meteorological professional questions and answers through review and screening.

7. A meteorological corpus construction system for a large meteorological service model, characterized in that: include: The format conversion module is used to collect the original corpus files in the meteorological field and convert the original corpus files into unified markdown format data; A data cleaning and structuring module is used to clean the markdown format data and organize it into a unified structured meteorological corpus; The data set construction module is used to combine the unified structured meteorological corpus with an external knowledge base to construct various data sets required for the meteorological service large model. The various data sets include data sets for pre-training, data sets for fine-tuning, data sets required for retrieval enhancement generation, and evaluation benchmark data sets.

8. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for constructing a meteorological corpus for a large meteorological service model as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for constructing a meteorological corpus for a large meteorological service model as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising computer instructions, characterized in that: The computer instructions instruct the computer to execute the method for constructing a meteorological corpus for a large meteorological service model as described in any one of claims 1-6.

Images