A large model customization training method and system for industry applications

Abstract

The application discloses a large model customization training method and system for industry applications, mainly relates to the technical field of large models, and is used to solve the problems that the prior art faces how to accurately divide terms, how to understand the hierarchical relationship in the protocol document, how to enhance the learning of the model on low-frequency technical points, and cannot effectively adapt to the characteristics of complex documents in the communication field, thereby causing unstable effects of the model in actual application. The application comprises the following steps: adopting a preset term importance mask as a bias term of attention weight of a large model, adjusting a preset professional term token in an attention score matrix to a preset promotion value, and reserving only top-k maximum attention values in each row; adopting a dynamic focus loss function to participate in adaptive loss calculation of parameters of the large model; iteratively training the large model to obtain a pre-trained large model; after pre-training is completed, adopting contrast learning for field adaptation to obtain a trained large model.

Description

Technical Field

[0001] The present application relates to the field of large-scale model training technology, and in particular to a large-scale model customized training method and system for industry applications. Background Art

[0002] With the rapid development of 5G and future 6G communication technologies, the communications industry needs to process massive amounts of data, including technical documents, protocol specifications, network logs, and user feedback. Efficiently and accurately extracting key information from this vast amount of information and implementing intelligent question-and-answering (Q&A) have become pressing technical challenges. Existing technologies mostly rely on traditional natural language processing methods, such as dictionary-based word segmentation, simple rule matching, and conventional Transformer models. However, these methods are less effective when dealing with the terminology, abbreviations, and hierarchical structures unique to the communications industry.

[0003] The text content in the communications industry is highly specialized and complex, with nested terminology, frequent abbreviations, and a hierarchical protocol stack structure, making traditional text parsing methods difficult to adapt. In this context, existing technologies generally face challenges in accurately segmenting terminology, understanding hierarchical relationships within protocol documents, and enhancing the model's learning of low-frequency technical points. Furthermore, existing model training methods mostly rely on static absolute position encoding and fixed word segmentation methods. These techniques are unable to effectively adapt to the characteristics of complex documents in the communications field, resulting in unstable model performance in practical applications. This is particularly true when processing protocol documents, diagnosing faults, and solving complex problems, where the model's performance often fails to meet actual needs. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, this application provides a large-scale model customized training method and system for industry applications to solve the problems faced by the prior art in how to accurately segment terms, how to understand the hierarchical relationship in the protocol document, how to enhance the model's learning of low-frequency technical points, and how to effectively adapt to the characteristics of complex documents in the communication field, which leads to unstable effects of the model in practical applications.

[0005] In a first aspect, the present application provides a large-scale model customized training method for industry applications, the method comprising:

[0006] Obtain original corpus from the communications industry through a pre-set training resource database; segment the original corpus, and obtain paragraph questions and paragraph answers from the segmented original paragraphs; and construct a question-and-answer corpus using paragraph questions and paragraph answers.

[0007] Input the question-answer corpus into a hierarchical decoding algorithm based on dynamic dictionary matching to obtain the preset optimal segmentation result for each input position in the question-answer corpus, and then complete the term segmentation of paragraph questions and paragraph answers in the question-answer corpus;

[0008] A semantic skeleton enhancement algorithm based on a dependency syntax tree samples and replaces the preset modification parameters in the paragraph questions in the question-answering corpus, constructs new paragraph questions with different expressions, and saves them in the question-answering corpus;

[0009] Based on the global position of each word in the paragraph question in the question-answering corpus and its relative position in the paragraph level, the large model performs weighted fusion to obtain position encoding. A preset term importance mask is used as the bias term for the large model's attention weight. The preset professional term token in the attention score matrix is ​​adjusted to a preset boost value, and only the top-k maximum attention values ​​are retained in each row. A dynamic focus loss function is used to participate in the large model parameter adaptive loss calculation. The large model is iteratively trained to obtain a pre-trained large model.

[0010] After pre-training is completed, contrastive learning is used for domain adaptation to obtain a trained large model.

[0011] In one implementation of the present application, after performing domain adaptation using contrastive learning to obtain a trained large model, the method further includes:

[0012] Problems in the communications industry where trained large models receive user input;

[0013] Perform term segmentation and dependency syntax analysis on the question to extract the preset core semantic skeleton;

[0014] Based on hierarchical dynamic position encoding and term-aware sparse attention, the preset core semantic skeleton is encoded into a preset high-dimensional semantic vector;

[0015] Generate the answer through the decoder;

[0016] The answer results are verified by the post-processing module, and statements that conflict with communication industry standards are eliminated, and finally the answer that complies with industry specifications is output.

[0017] In one implementation of the present application, the original corpus is segmented, and paragraph questions and paragraph answers are obtained from the segmented original paragraphs, specifically including:

[0018] With the help of preset rule templates, the original corpus is segmented, and paragraph questions and corresponding paragraph answers are obtained from the segmented original paragraphs.

[0019] In one implementation of the present application, a question-and-answer corpus is input into a hierarchical decoding algorithm based on dynamic dictionary matching to obtain a preset optimal segmentation result for each input position in the question-and-answer corpus, thereby completing the term segmentation of paragraph questions and paragraph answers in the question-and-answer corpus, specifically including:

[0020] The question-answer corpus is input into a hierarchical decoding algorithm based on dynamic dictionary matching. At each input position, a character substring of a preset length is searched backward; the preset length does not exceed the maximum segmentation length.

[0021] The language model generation probability of character substrings as candidate terms is calculated in sequence, and the priority scores of the character substrings in the communication industry terminology dictionary are obtained; and then the character substring with the highest comprehensive score is selected as the optimal segmentation result for the input position;

[0022] Among them, the optimal segmentation result of the entire input question-answer corpus is: through the formula:

[0023] , calculate the optimal segmentation result;

[0024] in, For the The optimal segmentation result of the input position; represents a positive integer;

[0025] represents the selection of candidate terms that maximize the objective function under constraints;

[0026] For the candidate segmentation terms, the search range is from the first The substring starting at the input position;

[0027] Indicates that the candidate term must belong to the communication industry term dictionary;

[0028] A dictionary of terminology for the communications industry;

[0029] is the maximum cutting length;

[0030] is the term priority coefficient;

[0031] is the conditional probability of the pre-trained language model;

[0032] The predecessor of the current candidate term A subsequence of characters;

[0033] The number to be predicted currently characters;

[0034] Represents the language model probability of generating the current character based on the prefix;

[0035] is an exponential function with a natural constant as its base;

[0036] is the priority ranking function for terms in the dictionary;

[0037] Representation term exist The priority number in ;

[0038] i is a positive integer used to quantify the optimal segmentation result of the i-th input position;

[0039] j is a positive integer used to quantify the jth candidate segmentation term.

[0040] In one implementation of the present application, a semantic skeleton enhancement algorithm based on a dependency syntax tree samples and replaces preset modification parameters in paragraph questions in a question-and-answer corpus, constructs new paragraph questions with different expressions, and saves them to the question-and-answer corpus. Specifically, the algorithm includes:

[0041] By performing dependency syntax analysis on paragraph questions, we identify the subject-verb-object core structure and the semantic skeleton of technical terms. We freeze the structure and replace it with a vocabulary of the same contextual parameters at the preset modification parameter nodes to construct new paragraph questions that are semantically coherent but have different expressions.

[0042] Among them, the formula for generating new paragraph questions is:

[0043] ,

[0044] Where, For new paragraph questions;

[0045] For the original paragraph question;

[0046] is the dependency syntax tree of Q;

[0047] It is the core term set extracted after term segmentation;

[0048] Indicates replacing the preset modified parameters in the syntax tree with words in the parameter vocabulary;

[0049] It is a parameter word list with the same context, representing a set of replaceable preset modifying parameter words in the same context.

[0050] In one implementation of the present application, based on the global position of each word in the paragraph question in the question-answer corpus and its relative position in the paragraph, the large model performs weighted fusion to obtain position encoding, specifically including:

[0051] By formula:

[0052] , obtain the position code;

[0053] in, is a positive integer, indicating the number of encoder layers of the large model;

[0054] For the The jth position encoding of the i-th token in the layer encoder;

[0055] k is a positive integer, representing the kth layer, and , is the total number of levels;

[0056] is the dynamic weight of the kth layer, and the calculation method is expressed as ;

[0057] is the Softmax function;

[0058] is the multi-layer perceptron mapping function;

[0059] is the hidden state representation of the i-1th position in the lth layer;

[0060] is the learnable weight vector corresponding to the k-th layer structure;

[0061] represents the standard sine-cosine position encoding in the kth layer;

[0062] The dimension corresponding to the standard sine-cosine position encoding in the kth layer;

[0063] It is a vector concatenation operation;

[0064] Indicates that based on paragraph The learnable embedding of relative positions is ;

[0065] is the dimension of the paragraph position embedding vector in the k-th layer.

[0066] In one implementation of this application, a preset term importance mask is used as a bias term for the attention weight of the large model, and the preset professional term token is adjusted to a preset boost value in the attention score matrix, and only the top-k maximum attention values ​​are retained in each row, specifically including:

[0067] By using the preset term importance mask as the bias term of the attention weight, the preset professional term token is adjusted to the preset boost value in the attention score matrix, and only the top-k maximum attention values ​​are retained in each row;

[0068] Through the attention weight matrix formula:

[0069] , calculate the attention weight matrix;

[0070] Where, is the attention weight matrix after sparseness;

[0071] Indicates that only the first The maximum value, Indicates the preset adjustment value;

[0072] is the query vector matrix;

[0073] is the key vector matrix;

[0074] for The transpose of

[0075] is the vector dimension;

[0076] Generating functions for diagonal matrices;

[0077] is the term importance mask vector, whose The elements are ,Right now, For the The mask value of a token is calculated as ;

[0078] For the candidate segmentation terms;

[0079] is a logarithmic function;

[0080] is the frequency of the term in the document;

[0081] is the inverse document frequency of the term in the corpus.

[0082] In one implementation of this application, a dynamic focus loss function is used to participate in the large model parameter adaptive loss calculation.

[0083] By formula: , calculate the dynamic focus loss;

[0084] Where, is the dynamic focus loss function;

[0085] is the total number of answer categories;

[0086] For the model The predicted probability of the class, where c∈[1,C];

[0087] is the category adaptive adjustment factor, and the calculation method is expressed as

[0088] ;

[0089] is the basic regulatory factor;

[0090] is the attenuation coefficient;

[0091] is the accuracy of the c-th category in the validation set;

[0092] Indicates the maximum accuracy of all categories in the validation set.

[0093] In one implementation of the present application, after pre-training is completed, contrastive learning is used to perform domain adaptation to obtain a trained large model, specifically including:

[0094] By formula:

[0095] , calculate the contrastive learning loss function;

[0096] Where, is the contrastive learning loss function;

[0097] Indicates a problem and the positive sample answer The semantic similarity score of

[0098] Indicates a problem and negative sample answers Similarity score of

[0099] is the temperature coefficient, and the calculation method is expressed as , is the initial value;

[0100] is the overall accuracy of the validation set;

[0101] is the preset positive sample answer;

[0102] The preset e-th negative sample answer, and e∈[1,E], E represents the number of negative samples in each batch.

[0103] In a second aspect, the present application provides a large-scale model customized training system for industry applications, the system comprising:

[0104] The construction module is used to obtain original corpus from the communications industry through a preset training resource database; segment the original corpus, obtain paragraph questions and paragraph answers from the segmented original paragraphs; and construct a question-and-answer corpus using the paragraph questions and paragraph answers;

[0105] The segmentation module is used to input the question and answer corpus into a hierarchical decoding algorithm based on dynamic dictionary matching to obtain the preset optimal segmentation result for each input position in the question and answer corpus, and then complete the term segmentation of paragraph questions and paragraph answers in the question and answer corpus;

[0106] The saving module is used to sample and replace the preset modification parameters in the paragraph questions in the question and answer corpus based on the semantic skeleton enhancement algorithm of the dependency syntax tree, construct new paragraph questions with different expressions, and save them to the question and answer corpus;

[0107] The training module is used to perform weighted fusion on the large model to obtain position encoding based on the global position of each word in the paragraph question in the question and answer corpus and the relative position in the hierarchical paragraph; a preset term importance mask is used as the bias item of the large model's attention weight, and the preset professional term token is adjusted to the preset improvement value in the attention score matrix, and only the Top-k maximum attention value is retained in each row; a dynamic focus loss function is used to participate in the large model parameter adaptive loss calculation; the large model is iteratively trained to obtain a pre-trained large model; after the pre-training is completed, contrastive learning is used for domain adaptation to obtain a trained large model.

[0108] Those skilled in the art will appreciate that this application has at least the following beneficial effects:

[0109] 1. Communications industry texts contain a large number of nested terms and abbreviations, which are difficult to effectively handle using traditional word segmentation methods. This invention combines language model probabilities with term priority to achieve multi-granular term segmentation, preventing incorrect term segmentation. This method demonstrates significant advantages in parsing complex communications terminology, particularly significantly improving the segmentation accuracy of multi-layered nested terms.

[0110] 2. To address the long-tail distribution problem in data in the communications field, this application adopts a semantic skeleton enhancement method based on a dependency syntax tree. By freezing core technical terms and replacing non-core parts to generate new samples, it ensures that the technical logic is not destroyed while expanding the coverage of the samples, effectively avoiding the semantic destruction problems that may be introduced in traditional data enhancement methods.

[0111] 3. The dynamic focus loss function proposed in this application can dynamically adjust the focus of the loss function to address problems with high similarity between long-tail data and incorrect answers, enhance the learning of low-frequency technical points, and strengthen the training of low-precision categories through adaptive adjustment factors. This significantly improves the model's discrimination ability, especially when dealing with complex faults and technical problems in the communications field.

[0112] 4. The traditional Transformer model uses fixed absolute position encoding, which is not applicable to the parsing of hierarchical protocol documents in the communications field. This application adopts hierarchical dynamic position encoding, which enables the model to better understand hierarchical structures such as protocol stacks, and enhances the modeling ability of key terms through the terminology-aware sparse attention mechanism, thereby improving the model's attention to professional terminology. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for the description. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0114] Figure 1 An embodiment of the present application provides a flow chart of a large-model customized training method for industry applications.

[0115] Figure 2 This is a three-dimensional surface diagram of the analytical performance of the present application under different term lengths and nesting levels provided in an embodiment of the present application.

[0116] Figure 3 This is a three-dimensional surface diagram of analytical performance under different traditional term lengths and nesting levels provided in an embodiment of the present application.

[0117] Figure 4 It is a three-dimensional surface diagram of a parameter optimization space of layered position coding provided by an embodiment of the present application.

[0118] Figure 5 This is a dual-axis trend chart of the optimization effect of a dynamic focal loss function on long-tail data distribution provided in an embodiment of the present application.

[0119] Figure 6This is a stacked area chart of a training process visualization comparison course learning strategy provided by an embodiment of the present application.

[0120] Figure 7 This is a schematic diagram of the internal structure of a large-model customized training system for industry applications provided in an embodiment of the present application. DETAILED DESCRIPTION

[0121] It should be understood by those skilled in the art that the embodiments described below are merely preferred embodiments of the present disclosure and do not imply that the present disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely intended to explain the technical principles of the present disclosure and are not intended to limit the scope of protection of the present disclosure. Based on the preferred embodiments provided by the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present disclosure.

[0122] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0123] The technical solutions proposed in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0124] The embodiment provides a large model customized training method for industry applications, such as Figure 1 As shown, the method provided in the embodiment of the present application mainly includes the following steps:

[0125] Step 110: Obtain original corpus of the communications industry through a preset training resource database; segment the original corpus, and obtain paragraph questions and paragraph answers from the segmented original paragraphs; and construct a question-and-answer corpus using the paragraph questions and paragraph answers.

[0126] It should be noted that this step may specifically include:

[0127] The communications industry's large-scale model training data mainly comes from a variety of structured and unstructured technical materials and interactive data, including standard documents, protocol specifications, technical white papers, operation and maintenance manuals, network logs, user problem feedback, and expert knowledge bases. For example, the 5G core network standards and "3GPP TS" protocol documents released by the Communications Standards Association, as well as the alarm logs and processing records recorded by an operator during actual operation and maintenance, can all be used as preset training resource databases for high-quality data sources; in addition, it also includes auxiliary materials such as manually constructed question-and-answer data, equipment fault location reports, and technical training handouts.

[0128] The data is acquired through a distributed data collection system, with crawlers and log collection components deployed in the cloud and edge devices. Structured information is retrieved through API interfaces and database linkage. For unstructured data, such as PDF, HTML, Word and other format files, text parsing and structure restoration are first performed, and named entity recognition and regular templates are combined to perform term extraction and content cleaning. After multiple rounds of preprocessing and content screening, a preset training resource database in the communications field is formed that can be used for subsequent sample construction.

[0129] Step 120: Input the question and answer corpus into a hierarchical decoding algorithm based on dynamic dictionary matching to obtain a preset optimal segmentation result for each input position in the question and answer corpus, thereby completing the term segmentation of paragraph questions and paragraph answers in the question and answer corpus.

[0130] For the acquired question-answer corpus, we first segment it into information units and convert the original paragraphs into question-answer pairs.

[0131] During the construction process, rule templates are used to extract question sentences and their corresponding descriptive answers from statements such as protocol definitions, process descriptions, parameter descriptions, and application scenarios;

[0132] For example, the question "What are the retransmission mechanisms of the PDCP layer?" and its answer are automatically generated from a description such as "The retransmission mechanism of the PDCP layer includes...";

[0133] For operational documents, you can also convert the operation steps into process questions and answers, such as "How to configure RRC connection reestablishment?"

[0134] At the same time, to ensure data quality and contextual consistency, the construction process uses a combination of manual review and automatic filtering to eliminate redundant, contradictory, or incomplete question-answer pairs;

[0135] When constructing question-answer pairs, it is also necessary to disambiguate polysemous words and abbreviations to ensure that the generated training samples conform to the expression standards of communication technology, and ultimately form a high-quality question-answer pair corpus suitable for Transformer architecture training.

[0136] Communications industry texts are characterized by dense professional terminology and complex nested abbreviations;

[0137] Conventional byte pair encoding tokenizers can easily cause term structure damage;

[0138] The present invention adopts a hierarchical decoding strategy based on dynamic dictionary matching, and realizes accurate segmentation of multi-granularity terms by fusing language model probability and term priority.

[0139] Specifically, at each input position, a character substring of a preset length is searched backward. The preset length can be defined as not exceeding the maximum segmentation length. The language model generation probability of each substring as a candidate term is calculated in turn. The substring with the highest comprehensive score is then selected as the optimal segmentation result for that position. The sliding window is then used to complete the term parsing of the entire input sequence, which can be expressed as:

[0140] ;

[0141] Where, For the The optimal segmentation result of the input position; represents a positive integer;

[0142] represents the selection of candidate terms that maximize the objective function under constraints;

[0143] For the candidate segmentation terms, the search range is from the first The substring starting at the input position;

[0144] Indicates that the candidate term must belong to the communication field term dictionary;

[0145] This is a communications terminology dictionary that includes standard terms such as "PDCP layer," "HARQ process," and "QCI level." It uses a predefined industry knowledge base to ensure that segmentation results comply with technical specifications. For example, it prevents common word segmenters from incorrectly segmenting embedded abbreviations (such as "NR" in "5G-NR," which stands for "New Radio") and compound terms (such as "HARQ process") in communications text.

[0146] The maximum segmentation length, for example, set to 8, The term length does not exceed the preset maximum. Dynamic window design is better than fixed length to prevent long terms from being broken into sub-segments. For example, "RadioResourceControl" may be as long as 21 characters, but the actual standardized abbreviation "RRC" is only 3 characters. Limiting the search scope balances efficiency and coverage;

[0147] is the term priority coefficient, which adjusts the weight of term priority and language model probability. For example, it is set to 0.3;

[0148] is the conditional probability of the pre-trained language model;

[0149] The predecessor of the current candidate term A subsequence of characters;

[0150] The number to be predicted currently characters;

[0151] Represents the language model probability of generating the current character based on the prefix;

[0152] is an exponential function with a natural constant as its base;

[0153] is the priority ranking function for terms in the dictionary;

[0154] Representation term exist The priority number in the ,e.g. "HARQ" has a higher priority than the general word "process";

[0155] i is a positive integer used to quantify the optimal segmentation result of the i-th input position;

[0156] j is a positive integer used to quantify the jth candidate segmentation term.

[0157] In one embodiment, an experimental verification analysis is performed.

[0158] 5,000 terms were extracted from 3GPP protocol documents and categorized by length (4-20 characters) and number of nesting levels (0-3 levels). The corresponding numbers are as follows:

[0159] Single-layer basic terms (such as "RLC"): 1,500;

[0160] Compound terms (e.g., "HARQ process"): 2,000;

[0161] Nested terms (e.g., “5G-NR carrier aggregation”): 1,500.

[0162] The control group of the experiment is the segmentation results of the BPE word segmenter and the maximum matching method (MaxMatch);

[0163] The three-dimensional surface analysis compares the parsing performance under different term lengths and nesting levels to verify the ability of the dynamic dictionary matching strategy to parse complex term structures in the communication field. The experiment uses term length and nesting levels as independent variables to compare the performance differences of traditional byte pair encoding methods under different complexity scenarios. Figure 2-Figure 3 It can be seen that as the term length increases and the nesting level deepens, the surface formed by this method is relatively flat and maintains a high position, while the surface of the traditional method shows a steep downward trend, especially in the long terms and deeply nested areas, where there is a clear depression. This reflects that the decoding strategy of the present invention integrates the language model probability and the term priority, which can effectively handle the hierarchical nesting relationship within the term, and avoid the structural destruction problem of atomic segmentation through dynamic weight balancing, thereby maintaining stable performance in complex term parsing scenarios.

[0164] In addition, to verify the adaptability of the multi-level term parsing algorithm to the complex term structure in the communications field, by comparing mainstream word segmentation technologies such as traditional byte pair encoding, word fragment segmentation, and maximum matching method, tests were carried out specifically on three typical scenarios: single-layer basic terms, compound terms, and nested terms. The experimental results show that in the nested term scenario, the traditional method has a significant decrease in accuracy due to its inability to identify cross-level semantic associations. However, this technology, through the fusion of dynamic dictionary matching and language model probability, demonstrates a strong structural preservation ability for compound terms and nested terms while ensuring the accuracy of basic term parsing. Each subgraph in the faceted scatter plot corresponds to a different term type, and the length of the error bar intuitively reflects the stability of the parsing. The standard deviation of this technology is the smallest under different term lengths, proving that its hierarchical decoding strategy can effectively alleviate the boundary ambiguity problem caused by the fixed segmentation rules of the traditional method.

[0165] Step 130: Based on the semantic skeleton enhancement algorithm of the dependency syntax tree, sample and replace the preset modification parameters in the paragraph questions in the question and answer corpus, construct new paragraph questions with different expressions, and save them in the question and answer corpus.

[0166] Communication question-answer pairs have a long-tail distribution problem, and there are insufficient samples of low-frequency technical points;

[0167] Conventional data augmentation methods such as synonym replacement can destroy technical semantics;

[0168] The present invention adopts semantic skeleton enhancement based on dependency syntax tree, keeping the technical subject unchanged and only replacing the modification parameters to generate new samples that conform to the technical logic;

[0169] Specifically, we perform dependency syntax analysis on the paragraph questions in the question-answer corpus, identify the semantic skeleton consisting of the subject-verb-object core structure and technical terms, freeze the structure unchanged, and sample and replace nodes such as non-core modifiers and parameter descriptions from the parameter vocabulary of the same context to construct new paragraph questions that are semantically coherent but have different expressions, while maintaining technical consistency while expanding sample coverage. This can be expressed as:

[0170] ;

[0171] Where, To enhance the generated paragraph questions;

[0172] is the original paragraph question text;

[0173] is the dependency syntax tree of question Q, representing the tree structure obtained after dependency syntax analysis of Q;

[0174] The core terminology set extracted after term segmentation represents the communication technology terminology nodes identified in the syntax tree. The technology entities (such as "LTE" and "protocol stack" in "LTE protocol stack") are identified through the dependency syntax tree, and their structure is frozen to avoid semantic damage.

[0175] Indicates that the modifying words other than terms in the syntax tree are replaced with words in the parameter vocabulary;

[0176] It is a parameter vocabulary with the same context, representing a set of replaceable parameter words in the same context. For example, "latency" can be replaced by "jitter" and "throughput", etc. It includes replaceable industry parameters, such as "latency → jitter" and "throughput → bandwidth". Replacement is limited to non-core modifiers. For example, "high" in "high latency" can be replaced by "severe".

[0177] Step 140: Based on the global position of each word in the paragraph question in the question and answer corpus and the relative position in the hierarchical paragraph, the large model performs weighted fusion to obtain the position encoding; a preset term importance mask is used as the bias item of the large model attention weight, and the preset professional term token is adjusted to a preset improvement value in the attention score matrix, and only the Top-k maximum attention value is retained in each row; a dynamic focus loss function is used to participate in the large model parameter adaptive loss calculation; the large model is iteratively trained to obtain a pre-trained large model; after the pre-training is completed, contrastive learning is used for domain adaptation to obtain a trained large model.

[0178] It's important to note that the tokens in this step can be understood as the "basic unit" of language in natural language processing, and can be words, subwords, or characters. Top-k is a term often used in text generation tasks in natural language processing. Its purpose is to select the k most likely tokens from the probability distribution predicted by the model when generating text, rather than simply selecting the token with the highest probability.

[0179] This application can use a large model based on the Transformer model architecture to build a knowledge question-answering system for the communications industry. Based on the conventional Transformer model architecture, improvements and training are performed. The process is as follows:

[0180] S401. Constructing a hierarchical dynamic position encoding mechanism:

[0181] The absolute position encoding of traditional Transformer cannot adapt to the hierarchical structure of communication documents, such as the layered description of protocol stacks;

[0182] The present invention adopts hierarchical dynamic position coding, which can distinguish the position relationship inside and outside the protocol layer when coding terms such as "RLCAM entity state machine";

[0183] Specifically, by adopting dual position information, namely the global position of a word in a sentence and its relative position in the hierarchical paragraph, and combining it with a learnable hierarchical embedding representation, the embedding fusion form not only retains the temporal information of the original Transformer, but also explicitly represents the position information in the protocol hierarchy, thereby improving the model's ability to understand structured documents, which can be expressed as:

[0184] , obtain the position code;

[0185] in, is a positive integer, indicating the number of encoder layers of the large model;

[0186] is a positive integer, representing the kth layer, and , is the total number of levels;

[0187] is a positive integer, which represents the index of the global tofken position;

[0188] is a positive integer, which represents the index of the relative position in the current paragraph;

[0189] For the The layer-wise value of the i-th token in the layer encoder Position code;

[0190] is the total number of levels, such as, Corresponding to the three layers of physical layer, transport layer and application layer;

[0191] is the dynamic weight of the kth layer, reflecting the contribution of the layer to the current position, and dynamically assigning weights to each layer. The calculation method is expressed as ;

[0192] is the Softmax function;

[0193] is the multi-layer perceptron mapping function;

[0194] For the The hidden state representation of the i-1th position in the layer is used to dynamically adjust the layer position weight;

[0195] is the learnable weight vector corresponding to the k-th layer structure;

[0196] represents the standard sine-cosine position encoding in the kth layer;

[0197] The dimension corresponding to the standard sine-cosine position encoding in the kth layer represents the Sinu parameter with decreasing wavelength;

[0198] It is a vector concatenation operation;

[0199] Indicates that based on paragraph The learnable embedding of relative positions is ;

[0200] is the dimension of the paragraph position embedding vector in the kth layer, such as setting The same as or set to its multiple to enhance the expressiveness of paragraph structure.

[0201] In one embodiment, an experimental verification analysis is performed.

[0202] The experiment used a set of protocol documents, specifically 100 documents including 3GPP TS 23.501 (5G system architecture), and manually annotated the layer boundaries (physical layer / transport layer / application layer);

[0203] The test tasks of the experiment are: protocol entity belonging judgment (e.g. “Which layer does the RLCAM entity belong to?”);

[0204] like Figure 4As shown in the figure, the parameter optimization space of layered position coding is explored through three-dimensional surface visualization, and the impact of the protocol layer perception mechanism on the model understanding ability is analyzed. The horizontal and vertical axes represent the number of coding layers and the concentration of the layer weight distribution, respectively. The surface height maps the protocol understanding accuracy. Experiments show that the traditional absolute position coding leads to performance saturation when the number of layers increases because it cannot distinguish the protocol layers. The dynamic weight allocation mechanism of this technology forms a high accuracy platform in the range of medium coding depth (8-10 layers) and moderate weight variance (0.3-0.4), proving that dual position coding can effectively capture the unique hierarchical structure characteristics of protocol documents. The special shape of the surface peak area confirms the typical three-layer architecture characteristics of the physical layer, transport layer, and application layer in the technical document, indicating that this encoding method has a strong representation ability for the communication protocol stack.

[0205] S402. Constructing a term-aware sparse attention mechanism:

[0206] Traditional multi-headed attention pays insufficient attention to professional terminology;

[0207] The present invention focuses on core terms based on sparse attention of term weights;

[0208] Specifically, by using the term importance mask as the bias term of the attention weight, the professional term token is moderately promoted in the attention score matrix, and only the top-k maximum attention values ​​are retained in each row, thereby sparsely sparsely focusing the attention structure. Different k value ratios are set for the term head and the grammatical head to highlight the semantic focus and improve the key term modeling capability, which is expressed as:

[0209] ;

[0210] Where, is the attention weight matrix after sparseness;

[0211] Indicates that only the first The maximum value, Indicates preset adjustment values, such as defining term headers , syntax head , N is the sequence length, Indicates rounding operation, the term head retains fewer but more critical connections, and the grammar head retains more context associations;

[0212] is the query vector matrix;

[0213] is the key vector matrix;

[0214] for The transpose of

[0215] is the vector dimension;

[0216] Generating functions for diagonal matrices;

[0217] is the term importance mask vector, whose The elements are ,Right now, For the The mask value of a token is calculated as In communication question-answering, there are many technical terms (such as "PDCP layer is responsible for header compression"), so the attention weight of common words (such as "responsible") needs to be suppressed. High-frequency terms (such as "HARQ") receive higher mask weights due to their high TF-IDF values, forcing the model to focus on the core technology.

[0218] It is a logarithmic function, and the default base is 10;

[0219] is the frequency of the term in the document;

[0220] is the inverse document frequency of the term in the corpus;

[0221] S403, calculate the dynamic focus loss function:

[0222] Aiming at the problem of long-tail data distribution and high similarity of wrong answers;

[0223] This paper adopts dynamic focus loss function to realize reinforcement learning of low-frequency technical points:

[0224] Specifically, by dynamically adjusting the adjustment factor based on the model's classification accuracy in different categories, we focus more on high-difficulty, low-precision categories and reduce the error rate of easily confused samples. This not only improves the model's ability to identify long-tail categories, but also effectively optimizes the model's learning performance for professional terminology, as expressed as:

[0225] ;

[0226] Where, is the dynamic focus loss function;

[0227] The total number of answer categories is calculated based on the standardized answer library in the communications field. The BERT semantic embedding method is used to convert technical descriptions into high-dimensional vectors. The hierarchical clustering algorithm is used to automatically aggregate answers with similar technical semantics into independent categories. The number of categories automatically generated when the clustering quality is optimal is used as the total number of answer categories. The value of

[0228] For the model The predicted probability of the class, where c∈[1,C];

[0229] is the category adaptive adjustment factor, and the calculation method is expressed as In communication technology support documents, there are many samples of common problems (such as "signal coverage"), but few samples of complex faults (such as "X2 interface abnormality in NSA networking"). Dynamic adjustment allows the model to balance learning priorities. The category adaptive adjustment factor dynamically increases the weight of categories with low validation set accuracy (such as the low-frequency fault type "RRC connection failure") to alleviate the long-tail distribution problem.

[0230] As the basic adjustment factor, for example, setting ;

[0231] For the attenuation coefficient, for example, set ;

[0232] is the accuracy of the c-th category in the validation set;

[0233] Indicates the maximum accuracy of all categories in the validation set.

[0234] In one embodiment, an experimental verification analysis is performed.

[0235] The long-tail dataset consists of 50,000 communication question-answer pairs, which are divided into five groups according to question frequency:

[0236]

[0237] like Figure 5 As shown in the figure, a dual-axis trend chart is used to analyze the optimization effect of the dynamic focus loss function on the long-tail data distribution, and the performance difference of the traditional cross entropy loss in different sample frequency ranges is compared. The left-axis curve shows that the method forms an obvious convex curve in the low-frequency sample area, while the traditional method shows a downward trend. The right-axis lifting curve shows that the maximum gain appears in the extremely low-frequency area, reflecting the core value of the adaptive adjustment factor. By dynamically amplifying the loss weight of the low-accuracy category, the model automatically focuses on low-frequency but key technical points such as "reasons for bearer establishment failure" in the technical documents during training, while avoiding overfitting of high-frequency conventional terms such as "channel quality indication", thereby improving the model's coverage of long-tail technical knowledge.

[0238] S404: Repeat the above steps until a preset stop iteration condition is met, indicating that the model training is complete. In one embodiment, the preset stop iteration condition is reaching a preset maximum number of iterations. Preferably, the preset maximum number of iterations is set to 1000 times.

[0239] S5. Implement a curriculum-based progressive training strategy:

[0240] In order to improve the efficiency of learning complex technical concepts, a four-stage course is designed, namely:

[0241] 1) Basic terminology stage: ;

[0242] Where, The technical complexity score of question Q is evaluated based on factors such as the depth of syntactic structure, the number of terms, and the length of logical chains;

[0243] Represents a set of training samples with lower technical complexity.

[0244] 2) Protocol interaction phase: ;

[0245] Where, For a specific protocol type ;

[0246] Represents a set of communication protocols, representing Long-Term-Evolution and New-Radio protocols respectively;

[0247] In the paired samples of question Q and answer A, the question involves LTE or NR protocol interaction content;

[0248] 3) Fault diagnosis stage: ;

[0249] Where, Represents a set of problem samples containing keywords such as "fault diagnosis", "fault analysis", and "problem troubleshooting".

[0250] 4) Scheme design stage: ;

[0251] Where, A training sample set representing a problem with high technical complexity, such as a training sample set with multi-layer nested structures or cross-segment logical reasoning.

[0252] During training, press Gradually expand the data set, and the learning rate of each stage decays to the previous stage times, while freezing the front of the underlying Transformer Layer parameter, L is the total number of layers.

[0253] In one embodiment, an experimental verification analysis is performed.

[0254] Set up a four-stage course set:

[0255]

[0256] like Figure 6 As shown in the figure, the differences in the effects of course learning strategies are visually compared through the training process. The stacked area chart shows the evolution of comprehensive capabilities in the four stages of basic terminology, protocol interaction, fault diagnosis, and solution design. The stage dividing line marks the course switching node. The traditional random sampling strategy shows obvious learning stagnation in the advanced stage. Although the fixed course strategy can achieve phased improvement, its generalization ability is insufficient in the later stage. The dynamic course strategy adopted by this technology is combined with the parameter freezing mechanism, which maintains a steady upward trend in the solution design stage. The changes in the area of ​​the color blocks in the graph reveal the contribution of each stage to the final performance. The dark green advantage area formed by this technology in the advanced stage intuitively reflects the ability of layered progressive training to gradually master complex technical concepts, and verifies that the course design conforms to the cognitive laws of the communication knowledge system.

[0257] After pre-training, contrastive learning is used for domain adaptation;

[0258] Specifically, by adopting a contrastive learning strategy to construct similar and dissimilar question-answer pairs, the model's discrimination ability in the communication field is improved by maximizing the similarity of positive samples and minimizing the similarity of negative samples. A dynamic temperature coefficient is used to gradually reduce the temperature as the verification accuracy improves, thereby improving the model's discrimination ability for difficult negative examples and achieving post-training optimization for domain semantic features, which can be expressed as:

[0259] ,

[0260] Where, is the contrastive learning loss function;

[0261] Indicates a problem and the positive sample answer The semantic similarity score of

[0262] Indicates a problem and negative sample answers Similarity score of

[0263] is the temperature coefficient, which is adjusted using an adaptive strategy. The calculation method is expressed as , As the initial value, set Traditional contrastive learning uses a fixed temperature, while dynamic adjustment is more suitable for the needs of gradually optimizing difficult examples in the communication field. The temperature coefficient is dynamically adjusted with the accuracy of the validation set to control the intensity of hard negative example mining, allowing the model to initially learn coarse-grained differences. As the accuracy improves, the temperature is gradually reduced to focus on hard negative examples.

[0264] is the overall accuracy of the validation set;

[0265] is the number of negative samples for each batch;

[0266] It should be noted that is a positive sample, such as the marked answer;

[0267] To preset the e-th negative sample answer, and e∈[1,E], E represents the number of negative samples in each batch, generated by:

[0268] a) Technical obfuscation samples: Replace key parameters of similar technologies, such as changing the "number of SSB beams" to the "number of CSI-RS ports";

[0269] b) Semantic adversarial examples: Generate grammatically correct but technically incorrect answers using BERT-MaskedLM, a text generation model based on a masked language model.

[0270] It should be noted that in the negative sample generation strategy, technical obfuscation samples replace similar technical parameters, such as "number of SSB beams → number of CSI-RS ports", to construct negative examples with similar semantics but technical errors; semantic adversarial samples use BERT-MaskedLM to generate grammatically correct but technically contradictory answers, such as changing "the PDCP layer is responsible for encryption" to "the RLC layer is responsible for encryption".

[0271] Industry integration point: In the communications field, the answers are highly similar (such as similar processes in different protocols), and the discrimination ability needs to be improved through refined negative examples.

[0272] The trained large model receives user input on communication problems, such as "How does the HARQ process implement the retransmission mechanism in 5G networks?"

[0273] First, the question is segmented into terms and subjected to dependency syntax analysis to extract the core semantic skeleton;

[0274] Then, the model encodes the question into a high-dimensional semantic vector based on hierarchical dynamic positional encoding and term-aware sparse attention, and gradually generates the answer through the decoder;

[0275] During the generation process, the model dynamically focuses on the key technical descriptions in the protocol document and combines the multi-level knowledge learned through course training to ensure the technical rigor of the answer.

[0276] The inference results are verified by the post-processing module, which eliminates statements that conflict with communication standards and ultimately outputs solutions that comply with industry specifications. This includes analyzing the interaction logic of each layer of the protocol stack in steps and associating relevant parameter configuration recommendations.

[0277] In actual deployment, the model supports both streaming response and batch processing modes;

[0278] For real-time question-answering scenarios, incremental decoding technology is used to prioritize the output of core conclusions on high-frequency technical points.

[0279] For complex solution design questions, the answer structure is optimized through multiple rounds of attention iteration to ensure the logical coherence and technical feasibility of the generated content;

[0280] At the same time, the inference engine has a built-in confidence assessment module. When a low-confidence term is detected, it automatically triggers the knowledge base retrieval and completion mechanism to avoid missing or incorrect transmission of key concepts.

[0281] S8. Implementation of cloud-edge collaborative architecture:

[0282] The cloud-edge collaborative architecture consists of a cloud training center and edge inference nodes;

[0283] High-performance computing clusters are deployed in the cloud to be responsible for global training, version management, and knowledge base updates of large models. Model parameters are synchronized to edge nodes using a distributed training framework.

[0284] On the edge side, lightweight inference engines are deployed in communication infrastructure such as base stations and core network equipment, loading large domain models that have been pruned and quantized to achieve low-latency localized question-answering services.

[0285] A secure channel is established between the cloud and the edge through a dedicated network, and model parameters and terminology dictionaries are synchronized regularly (such as updating 3GPP standard terminology) to ensure the consistency of the edge model and the cloud knowledge system.

[0286] At the resource scheduling level, the architecture adopts a dynamic load balancing strategy;

[0287] Simple, high-frequency questions (such as protocol parameter queries) are directly responded to by edge nodes. Complex tasks (such as cross-protocol fault diagnosis) are processed by edge nodes after semantic features are extracted and uploaded to the cloud for deep reasoning. The results are then compressed and transmitted back to the edge.

[0288] At the same time, anonymized interaction data (such as user question distribution) collected by edge nodes is fed back to the cloud to optimize course training strategies and dynamic focus loss functions;

[0289] In terms of privacy protection, sensitive data (such as internal operator protocols) is only processed at the edge, and non-sensitive data is desensitized before participating in cloud model iteration to achieve a balance between data value and security.

[0290] In addition, this application Figure 7 This application provides a large model customized training system for industry applications. Figure 7 As shown, the system provided in the embodiment of the present application mainly includes:

[0291] Construction module 210 is used to obtain original corpus of the communications industry from a preset training resource database; segment the original corpus, obtain paragraph questions and paragraph answers from the segmented original paragraphs; and construct a question-and-answer corpus using the paragraph questions and paragraph answers;

[0292] Segmentation module 220, for inputting the question-answer corpus into a hierarchical decoding algorithm based on dynamic dictionary matching, obtaining a preset optimal segmentation result for each input position in the question-answer corpus, and then completing term segmentation of paragraph questions and paragraph answers in the question-answer corpus;

[0293] A storage module 230 is configured to sample and replace preset modification parameters in paragraph questions in the question-answer corpus based on a semantic skeleton enhancement algorithm based on a dependency syntax tree, construct new paragraph questions with different expressions, and save them to the question-answer corpus;

[0294] The training module 240 is used to perform weighted fusion on the large model to obtain position encoding based on the global position of each word in the paragraph question in the question and answer corpus and the relative position in the hierarchical paragraph; a preset term importance mask is used as the bias item of the attention weight of the large model, and the preset professional term token is adjusted to a preset improvement value in the attention score matrix, and only the Top-k maximum attention value is retained in each row; a dynamic focus loss function is used to participate in the large model parameter adaptive loss calculation; the large model is iteratively trained to obtain a pre-trained large model; after the pre-training is completed, contrastive learning is used for domain adaptation to obtain a trained large model.

[0295] In addition, an embodiment of the present application also provides a non-volatile computer storage medium on which executable instructions are stored. When the executable instructions are executed, a large-model customized training method for industry applications as described above is implemented.

[0296] Thus far, the technical solutions of the present disclosure have been described in conjunction with the foregoing multiple embodiments. However, it is easy for those skilled in the art to understand that the scope of protection of the present disclosure is not limited to these specific embodiments. Without departing from the technical principles of the present disclosure, those skilled in the art may split and combine the technical solutions in the above-mentioned various embodiments, and may also make equivalent changes or replacements to the relevant technical features. Any changes, equivalent replacements, improvements, etc. made within the technical concepts and / or technical principles of the present disclosure will fall within the scope of protection of the present disclosure.

Claims

1. A large model customized training method for industry applications, characterized by: The method comprises: Obtain original corpus from the communications industry through a pre-set training resource database; segment the original corpus, and obtain paragraph questions and paragraph answers from the segmented original paragraphs; and construct a question-and-answer corpus using paragraph questions and paragraph answers. Input the question-answer corpus into a hierarchical decoding algorithm based on dynamic dictionary matching to obtain the preset optimal segmentation result for each input position in the question-answer corpus, and then complete the term segmentation of paragraph questions and paragraph answers in the question-answer corpus; The question-answer corpus is input into a hierarchical decoding algorithm based on dynamic dictionary matching, and a character substring of a preset length is searched backward at each input position; wherein the preset length does not exceed the maximum segmentation length; The language model generation probability of character substrings as candidate terms is calculated in sequence, and the priority scores of the character substrings in the communication industry terminology dictionary are obtained; and then the character substring with the highest comprehensive score is selected as the optimal segmentation result for the input position; Among them, the optimal segmentation result of the entire input question-answer corpus is: through the formula: , calculate the optimal segmentation result; among them, For the The optimal segmentation result of the input position; represents a positive integer; represents the selection of candidate terms that maximize the objective function under constraints; For the candidate segmentation terms, the search range is from the first The substring starting at the input position; Indicates that the candidate term must belong to the communication industry term dictionary; A dictionary of terminology for the communications industry; is the maximum cutting length; is the term priority coefficient; is the conditional probability of the pre-trained language model; The predecessor of the current candidate term A subsequence of characters; The number to be predicted currently characters; Represents the language model probability of generating the current character based on the prefix; is an exponential function with a natural constant as its base; is the priority ranking function for terms in the dictionary; Representation term exist The priority number in ; i is a positive integer used to quantify the optimal segmentation result of the i-th input position; j is a positive integer used to quantify the jth candidate segmentation term; The semantic skeleton enhancement algorithm based on the dependency syntax tree samples and replaces the preset modifying parameters in the paragraph questions in the question-answer corpus, constructs new paragraph questions with different expressions, and saves them in the question-answer corpus. By performing dependency syntax analysis on the paragraph questions, the semantic skeleton composed of the subject-verb-object core structure and technical terms is identified, the structure is frozen unchanged, and sampling and replacing are performed at the preset modifying parameter nodes from the vocabulary of the same context parameter to construct new paragraph questions with semantic coherence but different expressions. The formula for generating new paragraph questions is: , where For new paragraph questions; For the original paragraph question; for The dependency syntax tree of It is the core term set extracted after term segmentation; Indicates replacing the preset modified parameters in the syntax tree with words in the parameter vocabulary; The same-context parameter word list represents a set of replaceable preset modifying parameter words in the same context; Based on the global position of each word in the paragraph questions in the question-and-answer corpus and its relative position in the hierarchical paragraphs, the large model performs weighted fusion to obtain position encoding; a preset term importance mask is used as the bias item of the large model's attention weight, and the preset professional term token is adjusted to the preset improvement value in the attention score matrix, and only the Top-k maximum attention value is retained in each row; a dynamic focus loss function is used to participate in the large model parameter adaptive loss calculation; the large model is iteratively trained to obtain a pre-trained large model; after the pre-training is completed, contrastive learning is used for domain adaptation to obtain a trained large model.

2. The large-scale customized training method for industry applications according to claim 1 is characterized in that: After performing domain adaptation using contrastive learning to obtain a trained large model, the method further includes: Problems in the communications industry where trained large models receive user input; Perform term segmentation and dependency syntax analysis on the question to extract the preset core semantic skeleton; Based on hierarchical dynamic position encoding and term-aware sparse attention, the preset core semantic skeleton is encoded into a preset high-dimensional semantic vector; Generate the answer through the decoder; The answer results are verified by the post-processing module, and statements that conflict with communication industry standards are eliminated, and finally the answer that complies with industry specifications is output.

3. The large-scale customized training method for industry applications according to claim 1 is characterized in that: Segment the original corpus and obtain paragraph questions and paragraph answers from the segmented original paragraphs, including: With the help of preset rule templates, the original corpus is segmented, and paragraph questions and corresponding paragraph answers are obtained from the segmented original paragraphs.

4. The large-scale customized training method for industry applications according to claim 1 is characterized in that: Based on the global position of each word in the paragraph question in the question-answering corpus and its relative position in the paragraph level, the large model performs weighted fusion to obtain position encoding, specifically including: By formula: , obtain the position code; in, is a positive integer, indicating the number of encoder layers of the large model; For the The jth position encoding of the i-th token in the layer encoder; k is a positive integer, representing the kth layer, and , is the total number of levels; is the dynamic weight of the kth layer, and the calculation method is expressed as ; is the Softmax function; is the multi-layer perceptron mapping function; For the The hidden state representation of the i-1th position in the layer; is the learnable weight vector corresponding to the k-th layer structure; represents the standard sine-cosine position encoding in the kth layer; The dimension corresponding to the standard sine-cosine position encoding in the kth layer; It is a vector concatenation operation; Indicates that based on paragraph The learnable embedding of relative positions is ; is the dimension of the paragraph position embedding vector in the k-th layer.

5. The large-scale customized training method for industry applications according to claim 1 is characterized in that: The preset term importance mask is used as the bias term of the large model's attention weight. The preset professional term token is adjusted to the preset improvement value in the attention score matrix, and only the top-k maximum attention values ​​are retained in each row. Specifically, By using the preset term importance mask as the bias term of the attention weight, the preset professional term token is adjusted to the preset boost value in the attention score matrix, and only the top-k maximum attention values ​​are retained in each row; Through the attention weight matrix formula: , calculate the attention weight matrix; Where, is the attention weight matrix after sparseness; Indicates that only the first The maximum value, Indicates the preset adjustment value; is the query vector matrix; is the key vector matrix; for The transpose of is the vector dimension; Generating functions for diagonal matrices; is the term importance mask vector, whose The elements are ,Right now, For the The mask value of a token is calculated as ; For the candidate segmentation terms; is a logarithmic function; is the frequency of the term in the document; is the inverse document frequency of the term in the corpus.

6. The large-scale customized training method for industry applications according to claim 1 is characterized in that: Adopting dynamic focus loss function, participating in large model parameter adaptive loss calculation, By formula: , calculate the dynamic focus loss; Where, is the dynamic focus loss function; is the total number of answer categories; For the model The predicted probability of the class, where c∈[1,C]; is the category adaptive adjustment factor, and the calculation method is expressed as ; is the basic regulatory factor; is the attenuation coefficient; is the accuracy of the c-th category in the validation set; Indicates the maximum accuracy of all categories in the validation set.

7. The large-scale customized training method for industry applications according to claim 1 is characterized in that: After pre-training is completed, contrastive learning is used for domain adaptation to obtain a trained large model, which includes: By formula: , calculate the contrastive learning loss function; Where, is the contrastive learning loss function; Indicates a problem and the positive sample answer The semantic similarity score of Indicates a problem and negative sample answers Similarity score of is the temperature coefficient, and the calculation method is expressed as , is the initial value; is the overall accuracy of the validation set; is the preset positive sample answer; The preset e-th negative sample answer, and e∈[1,E], E represents the number of negative samples in each batch.

8. A large-scale model customized training system for industry applications, characterized by: The system comprises: The construction module is used to obtain original corpus from the communications industry through a preset training resource database; segment the original corpus, obtain paragraph questions and paragraph answers from the segmented original paragraphs; and construct a question-and-answer corpus using the paragraph questions and paragraph answers; A segmentation module is used to input the question and answer corpus into a hierarchical decoding algorithm based on dynamic dictionary matching, obtain a preset optimal segmentation result for each input position in the question and answer corpus, and then complete the term segmentation of paragraph questions and paragraph answers in the question and answer corpus; the question and answer corpus is input into the hierarchical decoding algorithm based on dynamic dictionary matching, and backward searches for a character substring of a preset length at each input position; wherein the preset length does not exceed the maximum segmentation length; The language model generation probability of character substrings as candidate terms is calculated in sequence, and the priority scores of the character substrings in the communication industry terminology dictionary are obtained; and then the character substring with the highest comprehensive score is selected as the optimal segmentation result for the input position; Among them, the optimal segmentation result of the entire input question-answer corpus is: through the formula: , calculate the optimal segmentation result; among them, For the The optimal segmentation result of the input position; represents a positive integer; represents the selection of candidate terms that maximize the objective function under constraints; For the candidate segmentation terms, the search range is from the first The substring starting at the input position; Indicates that the candidate term must belong to the communication industry term dictionary; A dictionary of terminology for the communications industry; is the maximum cutting length; is the term priority coefficient; is the conditional probability of the pre-trained language model; The predecessor of the current candidate term A subsequence of characters; The number to be predicted currently characters; Represents the language model probability of generating the current character based on the prefix; is an exponential function with a natural constant as its base; is the priority ranking function for terms in the dictionary; Representation term exist The priority number in ; i is a positive integer used to quantify the optimal segmentation result of the i-th input position; j is a positive integer used to quantify the jth candidate segmentation term; The saving module is used to sample and replace the preset modification parameters in the paragraph questions in the question-answering corpus using a semantic skeleton enhancement algorithm based on the dependency syntax tree, construct new paragraph questions with different expressions, and save them to the question-answering corpus. By performing dependency syntax analysis on the paragraph questions, the semantic skeleton composed of the subject-verb-object core structure and technical terms is identified, the structure is frozen unchanged, and sampling and replacing are performed from the same context parameter vocabulary at the preset modification parameter nodes to construct new paragraph questions with semantic coherence but different expressions. The formula for generating the new paragraph question is: , where For new paragraph questions; For the original paragraph question; for The dependency syntax tree of It is the core term set extracted after term segmentation; Indicates replacing the preset modified parameters in the syntax tree with words in the parameter vocabulary; The same-context parameter word list represents a set of replaceable preset modifying parameter words in the same context; The training module is used to perform weighted fusion on the large model to obtain position encoding based on the global position of each word in the paragraph question in the question and answer corpus and the relative position in the hierarchical paragraph; a preset term importance mask is used as the bias item of the large model's attention weight, and the preset professional term token is adjusted to the preset improvement value in the attention score matrix, and only the Top-k maximum attention value is retained in each row; a dynamic focus loss function is used to participate in the large model parameter adaptive loss calculation; the large model is iteratively trained to obtain a pre-trained large model; after the pre-training is completed, contrastive learning is used for domain adaptation to obtain a trained large model.

Images