A method and device for data governance of large model fine-tuning corpus
By performing data governance on the corpus for fine-tuning large language models, including segmentation, vectorization, and importance ranking, the problems of large corpus quantity and high repetition were solved, the corpus quality was improved, the annotation difficulty was reduced, and the fine-tuning effect was guaranteed.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA UNITECHS
- Filing Date
- 2023-12-12
- Publication Date
- 2026-07-03
AI Technical Summary
The large language model fine-tuning corpus has problems such as huge quantity, high repetition and great difficulty in annotation, which makes the annotation work difficult to advance and requires the participation of professionals.
Before fine-tuning, the corpus undergoes data governance, including segmentation, vectorization, similarity analysis, and importance ranking, to select the highest priority corpus segments for annotation.
It improved the quality of the corpus, reduced the difficulty of annotation, ensured the effect of fine-tuning, and improved the annotation efficiency.
Smart Images
Figure CN118012827B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data governance, and in particular to a method and apparatus for data governance of large model fine-tuning corpora. Background Technology
[0002] Large Language Models (LLMs) are models with a massive number of parameters pre-trained on vast corpora, possessing the ability to generate thought chains and content. Based on these LLMs, further targeted fine-tuning can be performed using annotated corpora from a specific industry, making the LLMs more proficient in the domain knowledge of that industry and better suited to the needs of industry applications.
[0003] The effectiveness of LLM fine-tuning largely depends on the quality of the text corpus annotation. However, the current corpus accumulation in various industries clearly exhibits three characteristics: First, the quantity is enormous, potentially reaching tens or hundreds of thousands of documents; second, the repetition rate is high, with many documents containing highly similar or even identical content fragments; and third, annotation is difficult and labor-intensive. Annotating industry-specific corpora requires personnel with a deep understanding of the industry background and business characteristics to accurately annotate them, but such personnel often have very full daily workloads and find it difficult to dedicate much time to annotation work.
[0004] It is evident that among the three main characteristics mentioned above, characteristics one, two, and three represent the primary points of contention. On the one hand, the corpus is massive in quantity and has a high degree of repetition; on the other hand, it places high demands on the annotators, making the annotation work difficult to implement. Therefore, before initiating the annotation work, we need to perform data governance on the corpus, extract key information from it, and analyze its priorities and importance in order to rationally plan the annotation process. Summary of the Invention
[0005] To address the problems existing in the prior art, this invention provides a method and apparatus for data governance of large model fine-tuning corpora. Before fine-tuning the large model, the corpus is pre-processed with data governance to improve the quality of the corpus, reduce the difficulty of annotation, and ensure the fine-tuning effect.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] In one embodiment of the present invention, a method for data governance of large model fine-tuning corpora is proposed, the method comprising:
[0008] S01. Collect document corpora, store them centrally, and mark each document to indicate whether segmentation and corpus analysis have been completed during storage;
[0009] S02. Perform segmentation operation on each segment of all the corpus to be segmented;
[0010] In step S02, a corresponding segmentation scheme is used based on the document type, specifically including:
[0011] S021. Documents of code type should be segmented according to the code structure, such as each function or each class should be segmented into a piece.
[0012] S022. Documents with specific content format descriptions, such as Markdown and XML, are segmented into sections based on content paragraphs;
[0013] S023. For WORD and PDF documents, first divide them into segments of fixed length, and then perform operations such as merging, splitting, shrinking, and expanding the segments based on document similarity.
[0014] S024. For documents with rigid formatting requirements, such as network configuration scripts, a customized segmentation scheme should be developed based on the document characteristics. For example, if a manufacturer's network equipment uses the hash symbol "#" as the segment separator in its configuration files, then the segmentation should be performed directly according to the manufacturer's specified separator.
[0015] S03. Embedding and vectorizing the segmented document fragments.
[0016] Furthermore, the vectorization methods in S03 include: word2vec, FastTex t, wordRank, or can be implemented by coding yourself;
[0017] Embedding is a term in deep learning that refers to a series of matrix construction and dimensionality adjustment operations that convert text into a data-based model that can perform mathematical calculations.
[0018] Word2vec, FastText, and wordRank are all mature solutions commonly used in the industry for implementing embedding operations.
[0019] S04. The vectorized information is stored in the vector library;
[0020] Furthermore, the vector library selection in S04 includes: Chroma, FAISS, Pine cone, etc.
[0021] S05. Perform similarity analysis on the segmented document fragments and mark the document fragments with high similarity as duplicate document fragments;
[0022] Furthermore, the similarity analysis scheme used in S05 includes TF-IDF, BM25, etc., or can be implemented by self-coding.
[0023] TF-IDF and BM25 are both mature solutions commonly used in the industry for similarity analysis.
[0024] S06. Count the number of repetitions for each document segment and temporarily label this data as n;
[0025] Furthermore, the repetition count in S06 uses a similarity algorithm to determine whether the numbers are similar based on a set threshold.
[0026] S07. Perform deduplication on the document fragment and count the number of subsequent branches, temporarily labeling the number of subsequent branches of the document fragment as p;
[0027] Furthermore, the following branch in S07 is: in the original document corpus, the next document segment immediately following the current document segment.
[0028] Furthermore, the subsequent branch statistics in S07 include:
[0029] S071. A repetitive document fragment, recording a branch;
[0030] S072. Record the number of branches layer by layer, and perform a weighted calculation on the number of branches at each layer;
[0031] S073. For each document segment, sum up the weighted results of the subsequent branches at each level to obtain the number of subsequent branches for that document segment, and temporarily label this data as p.
[0032] S08. Sort the document fragments by importance based on the repetition degree n and the number of subsequent branches p of each document fragment;
[0033] Further, S08 includes:
[0034] S081. For large model fine-tuning scenarios that favor configuration command generation, code generation, etc., calculate the importance index K based on p and set the formula.
[0035] The formula can be set freely. The goal is to sort and filter out the corpus segments that are expected to be marked first during fine-tuning by using importance indicators, based on the needs of the business scenario and the characteristics of the document corpus itself.
[0036] S082. For large-scale model fine-tuning scenarios that are biased towards knowledge-based question answering and document-assisted generation, the importance index K is calculated with n as the main component.
[0037] S083. Sort the document fragments according to the importance index K.
[0038] S09. Convert the vectorized document fragments into text for use by corpus annotators.
[0039] In one embodiment of the present invention, a device for data governance of large model fine-tuning corpora is also proposed, the device comprising:
[0040] The corpus collection module collects document corpora, stores them centrally, and marks each document during storage to indicate whether segmentation and corpus analysis have been completed.
[0041] The document fragment segmentation module performs fragment segmentation operations on each of the corpora to be segmented.
[0042] The document fragment vectorization module performs embedding vectorization on the segmented document fragments.
[0043] The data entry module stores the vectorized information into the vector library.
[0044] The similarity analysis module performs similarity analysis on the segmented document fragments and marks document fragments with high similarity as duplicate document fragments;
[0045] The document fragment duplication statistics module counts the number of times each document fragment is repeated, and this data is temporarily labeled as n.
[0046] The post-branch statistics module deduplicates document fragments and counts the number of post-branch segments, temporarily labeling the number of post-branch segments as p.
[0047] The document fragment importance ranking module ranks document fragments based on their repetition degree n and the number of subsequent branches p.
[0048] The corpus output module converts vectorized document fragments into text for use by corpus annotators.
[0049] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for data governance of large model fine-tuning corpora.
[0050] In one embodiment of the present invention, a computer-readable storage medium is also proposed, which stores a computer program that performs a method for data governance of a large model fine-tuning corpus.
[0051] Beneficial effects:
[0052] This invention provides a method and apparatus for data governance of corpora for fine-tuning large models. The method involves analyzing the corpus, selecting the highest priority corpus segments, and delivering them to annotators for annotation. The corpus is deduplicated, and for corpus segments with high similarity, the annotators only need to annotate them once. Before fine-tuning the large model, the corpus undergoes pre-processing data governance to improve corpus quality, reduce annotation difficulty, and ensure the effectiveness of fine-tuning. Attached Figure Description
[0053] Figure 1 This is a schematic diagram of the data governance method for fine-tuning large model corpora according to the present invention;
[0054] Figure 2 This is a schematic diagram of the device structure for data governance of large model fine-tuning corpora according to the present invention;
[0055] Figure 3 This is a schematic diagram of a computer device structure according to an embodiment of the present invention. Detailed Implementation
[0056] The principles and spirit of the present invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.
[0057] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.
[0058] The technical terms involved in this invention and their explanations are as follows:
[0059] Embedding is a term in deep learning that refers to a series of matrix construction and dimensionality adjustment operations that convert text into a data-based model that can perform mathematical calculations.
[0060] Word2vec, FastText, and wordRank are all mature solutions commonly used in the industry for implementing embedding operations.
[0061] TF-IDF and BM25 are both mature solutions commonly used in the industry for similarity analysis.
[0062] According to an embodiment of the present invention, a method and apparatus for data governance of large model fine-tuning corpus are proposed. Before fine-tuning the large model, the corpus is pre-processed to improve the quality of the corpus, reduce the difficulty of annotation, and ensure the fine-tuning effect.
[0063] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.
[0064] like Figure 1 As shown, the present invention relates to a method for data governance of large model fine-tuning corpora, the method comprising:
[0065] S01. Collect document corpora, store them centrally, and mark each document to indicate whether segmentation and corpus analysis have been completed during storage;
[0066] S02. Perform segmentation operation on each segment of all the corpus to be segmented;
[0067] In step S02, a corresponding segmentation scheme is used based on the document type, specifically including:
[0068] S021. Documents of code type should be segmented according to the code structure, such as each function or each class should be segmented into a piece.
[0069] S022. Documents with specific content format descriptions, such as Markdown and XML, are segmented into sections based on content paragraphs;
[0070] S023. For WORD and PDF documents, first divide them into segments of fixed length, and then perform operations such as merging, splitting, shrinking, and expanding the segments based on document similarity.
[0071] S024. For documents with rigid formatting requirements, such as network configuration scripts, a customized segmentation scheme should be developed based on the document characteristics. For example, if a manufacturer's network equipment uses the hash symbol "#" as the segment separator in its configuration files, then the segmentation should be performed directly according to the manufacturer's specified separator.
[0072] S03. Embedding and vectorizing the segmented document fragments.
[0073] Furthermore, the vectorization methods in S03 include: word2vec, FastTex t, wordRank, or can be implemented by coding yourself;
[0074] Embedding is a term in deep learning that refers to a series of matrix construction and dimensionality adjustment operations that convert text into a data-based model that can perform mathematical calculations.
[0075] Word2vec, FastText, and wordRank are all mature solutions commonly used in the industry for implementing embedding operations.
[0076] S04. The vectorized information is stored in the vector library;
[0077] Furthermore, the vector library selection in S04 includes: Chroma, FAISS, Pine cone, etc.
[0078] S05. Perform similarity analysis on the segmented document fragments and mark the document fragments with high similarity as duplicate document fragments;
[0079] Furthermore, the similarity analysis scheme used in S05 includes TF-IDF, BM25, etc., or can be implemented by self-coding.
[0080] TF-IDF and BM25 are both mature solutions commonly used in the industry for similarity analysis.
[0081] S06. Count the number of repetitions for each document segment and temporarily label this data as n;
[0082] Furthermore, the repetition count in S06 uses a similarity algorithm to determine whether the numbers are similar based on a set threshold.
[0083] S07. Perform deduplication on the document fragment and count the number of subsequent branches, temporarily labeling the number of subsequent branches of the document fragment as p;
[0084] Furthermore, the following branch in S07 is: in the original document corpus, the next document segment immediately following the current document segment.
[0085] Furthermore, the subsequent branch statistics in S07 include:
[0086] S071. A repetitive document fragment, recording a branch;
[0087] S072. Record the number of branches layer by layer, and perform a weighted calculation on the number of branches at each layer;
[0088] S073. For each document segment, sum up the weighted results of the subsequent branches at each level to obtain the number of subsequent branches for that document segment, and temporarily label this data as p.
[0089] To make it easier to understand, here is an additional illustration of a rear branch.
[0090] If there are currently three documents:
[0091] After document 1 is segmented, its content is <fragment a><fragment b><fragment c><fragment d>;
[0092] After document 2 is segmented, its content is <fragment t><fragment a><fragment e>;
[0093] After document 3 is segmented, its content is <fragment x><fragment a><fragment b><fragment y>;
[0094] As you can see, for <fragment a>, there may be two subsequent fragments, <fragment b> and <fragment e>, where <fragment b> will appear repeatedly. This is the first-level trailing branch.
[0095] Following the combination of <fragment a> and <fragment b>, there will be two more fragments, <fragment c> and <fragment y>, which are the second-level appendage branches.
[0096] S08. Sort the document fragments by importance based on the repetition degree n and the number of subsequent branches p of each document fragment;
[0097] Further, S08 includes:
[0098] S081. For large model fine-tuning scenarios that favor configuration command generation, code generation, etc., calculate the importance index K based on p and set the formula.
[0099] The formula can be set freely. The goal is to sort and filter out the corpus segments that are expected to be marked first during fine-tuning by using importance indicators, based on the needs of the business scenario and the characteristics of the document corpus itself.
[0100] S082. For large-scale model fine-tuning scenarios that are biased towards knowledge-based question answering and document-assisted generation, the importance index K is calculated with n as the main component.
[0101] S083. Sort the document fragments according to the importance index K.
[0102] S09. Convert the vectorized document fragments into text for use by corpus annotators.
[0103] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0104] To provide a clearer explanation of the above-described method for data governance of large model fine-tuning corpora, specific embodiments are described below. However, it is worth noting that these embodiments are only for better illustrating the present invention and do not constitute an improper limitation of the present invention.
[0105] In a scenario involving fine-tuning a large model, the plan is to use the configuration files of network devices as the main corpus. After annotation and fine-tuning, a large model that can be generated with the assistance of configuration commands will be produced.
[0106] S01. Collect configuration files from various network equipment manufacturers, such as Huawei, ZTE, H3C, Fiberhome, etc., with 100 configuration files per manufacturer.
[0107] S02. Segment the configuration file according to the delimiter specified by the manufacturer, use TF-IDF to identify the keywords in the configuration file, and identify the remaining characters as variables;
[0108] S03. Convert variables in the configuration file to fixed variable names, such as converting IPs like 1.1.1.1 and 2.2.2.2 to $IP, and converting time-related variables to $DATE, $TIME, etc.
[0109] S04. The vectorized information is stored in the vector library;
[0110] S05. Perform similarity analysis on the segmented document fragments and mark the document fragments with high similarity as duplicate document fragments.
[0111] S06. Calculate the repetition degree n for each document fragment after the above processing;
[0112] S07. Calculate the subsequent branch p for each document segment;
[0113] During the calculation, only the next three layers of segments are calculated;
[0114] The first-level segment has its branch count fully calculated and denoted as T1.
[0115] For the second layer segment, the number of branches is weighted. Assuming the actual number of branches in this layer is t2, the number of branches used for calculation after weighting is T2, and T2 = (t2-1)*0.5;
[0116] For the third layer segment, the number of branches is weighted. Assuming the actual number of branches in this layer is t3, the number of branches used for calculation after weighting is T3, and T3 = (t3-1)*0.1;
[0117] The number of subsequent branches p in this document fragment is calculated using the formula p = T1 + T2 + T3;
[0118] S08. Calculate the importance index K for all document fragments, K = P * 2 + n
[0119] S09. Sort all document fragments in reverse order according to K, and retrieve the Top N document fragments from the vector library each time for use by the annotators.
[0120] Based on the same inventive concept, this invention also proposes an apparatus for data governance of large model fine-tuning corpora. The implementation of this apparatus can refer to the implementation of the methods described above, and repeated details will not be repeated. The term "module" used below can refer to a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0121] Figure 2 This is a schematic diagram of the device structure for data governance of large-scale model fine-tuning corpora according to the present invention. (For example...) Figure 2 As shown, the device includes:
[0122] The corpus collection module 110 collects document corpora, stores them centrally, and marks each document as having completed segmentation and corpus analysis during storage.
[0123] The document fragment segmentation module 120 performs fragment segmentation operations on each of the corpora to be segmented.
[0124] The document fragment vectorization module 130 performs embedding vectorization on the segmented document fragments;
[0125] The data entry module 140 stores the vectorized information into the vector library.
[0126] The similarity analysis module 150 performs similarity analysis on the segmented document fragments and marks document fragments with high similarity as duplicate document fragments;
[0127] The document fragment duplication statistics module 160 counts the number of times each document fragment is repeated, and this data is temporarily labeled as n.
[0128] The post-branch statistics module 170 performs deduplication on document fragments and counts the number of post-branch segments, temporarily labeling the number of post-branch segments of a document fragment as p;
[0129] The document fragment importance ranking module 180 ranks document fragments according to their repetition degree n and the number of subsequent branches p.
[0130] The corpus output module 190 converts vectorized document fragments into text for use by corpus annotators.
[0131] It should be noted that although several modules of the apparatus for data governance of large model fine-tuning corpora are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.
[0132] Based on the aforementioned inventive concept, such as Figure 3 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, it implements the aforementioned method for data governance of large model fine-tuning corpus.
[0133] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that performs the aforementioned method for data governance of large model fine-tuning corpora.
[0134] This invention provides a method and apparatus for data governance of corpora for fine-tuning large models. The method involves analyzing the corpus, selecting the highest priority corpus segments, and delivering them to annotators for annotation. The corpus is deduplicated, and for corpus segments with high similarity, the annotators only need to annotate them once. Before fine-tuning the large model, the corpus undergoes pre-processing data governance to improve corpus quality, reduce annotation difficulty, and ensure the effectiveness of fine-tuning.
[0135] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
[0136] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0137] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0138] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0140] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0141] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0142] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0143] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
[0144] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.
Claims
1. A method for data governance on large model fine-tuning corpus, characterized in that, The method includes: S01. Collect document corpora, store them centrally, and mark each document to indicate whether segmentation and corpus analysis have been completed during storage; S02. Perform segmentation operation on each segment of all the corpus to be segmented; S03. Embedding vectorizes the segmented document fragments; S04. The vectorized information is stored in the vector library; S05. Perform similarity analysis on the segmented document fragments and mark the document fragments with high similarity as duplicate document fragments; S06. Count the number of repetitions for each document segment and temporarily label this data as n; S07. Perform deduplication on the document fragment and count the number of subsequent branches, temporarily labeling the number of subsequent branches of the document fragment as p; The following branch is: in the original document corpus, the next document segment immediately following the current document segment; The statistics for subsequent branching include: S071. A repetitive document fragment, recording a branch; S072. Record the number of branches layer by layer, and perform a weighted calculation on the number of branches at each layer; S073. For each document segment, sum up the weighted results of the subsequent branches at each level to obtain the number of subsequent branches for that document segment, and temporarily label this data as p; S08. Rank the document fragments by importance based on their repetition degree n and the number of subsequent branches p, including: S081. For large model fine-tuning scenarios that favor configuration command generation and code generation, use p as the main factor, calculate the importance index K, and set the formula. S082. For large-scale model fine-tuning scenarios that favor knowledge-based question answering and document-assisted generation, the importance index K is calculated with n as the main component. S083. Sort the document fragments according to the importance index K; S09. Convert the vectorized document fragments into text for use by corpus annotators.
2. The method for data governance of large model fine-tuning corpora according to claim 1, characterized in that, The step S02, which uses a corresponding segmentation scheme based on the document type, includes: S021. Documents of a code nature are segmented according to the code structure; S022. Documents with accompanying content format descriptions are divided into segments based on content paragraphs; S023. For WORD and PDF documents, first divide them into segments of fixed length, and then merge, split, shrink, and expand the segments based on document similarity. S024. For documents with rigid format requirements, a customized segmentation scheme shall be developed based on the document characteristics.
3. The method for data governance of large model fine-tuning corpora according to claim 1, characterized in that, The vectorization methods in S03 include word2vec, FastText, wordRank, or can be implemented by coding yourself.
4. The method for data governance of large model fine-tuning corpora according to claim 1, characterized in that, The vector library selections in S04 include: Chroma, FAISS, and Pinecone.
5. The method for data governance of large model fine-tuning corpora according to claim 1, characterized in that, The similarity analysis schemes used in S05 include TF-IDF, BM25, or self-coded implementations.
6. The method for data governance of large model fine-tuning corpora according to claim 1, characterized in that, The repetition count in S06 uses a similarity algorithm to determine whether the numbers are similar based on a set threshold.
7. A device for data governance of large model fine-tuning corpora, characterized in that, The device includes: The corpus collection module collects document corpora, stores them centrally, and marks each document during storage to indicate whether segmentation and corpus analysis have been completed. The document fragment segmentation module performs fragment segmentation operations on each of the corpora to be segmented. The document fragment vectorization module performs embedding vectorization on the segmented document fragments. The data entry module stores the vectorized information into the vector library. The similarity analysis module performs similarity analysis on the segmented document fragments and marks document fragments with high similarity as duplicate document fragments; The document fragment duplication statistics module counts the number of times each document fragment is repeated, and this data is temporarily labeled as n. The post-branch statistics module deduplicates document fragments and counts the number of post-branch segments, temporarily labeling the number of post-branch segments as p. The following branch is: in the original document corpus, the next document segment immediately following the current document segment; The statistics for subsequent branching include: Repetitive document fragments, recording a branch; Record the number of branches layer by layer, and calculate the number of branches at each layer using weighted averages. For each document segment, sum up the weighted results of the subsequent branches at each level to obtain the number of subsequent branches for that document segment, and temporarily label this data as p; The document fragment importance ranking module ranks document fragments based on their repetition degree *n* and the number of subsequent branches *p*, including: For large model fine-tuning scenarios that favor configuration command generation and code generation, we use p as the main factor, calculate the importance index K, and set the formula accordingly. For large-scale model fine-tuning scenarios that lean towards knowledge-based question answering and document-assisted generation, the importance index K is calculated with n as the main component. Sort document fragments according to the importance metric K; The corpus output module converts vectorized document fragments into text for use by corpus annotators.
8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can run on a processor, the processor executing the computer program of the method according to any one of claims 1-6.