A core training set construction method, a large model training method, and related devices

Abstract

This invention discloses a core training set construction method, a large model training method, and related apparatus, relating to the field of large model training technology. The core training set construction method includes: employing an iterative screening approach to select samples from a candidate sample set and add them to an initially empty first core training set until an iteration termination condition is met. The iteration termination condition is that after adding samples in the most recent iteration, the semantic coverage gain of the first core training set is less than a preset coverage gain threshold. Then, based on the semantic distance between samples, semantically non-redundant samples are selected from the first core training set to construct a second core training set. This second core training set is used to train a large model. The core training set construction method provided by this invention can construct a core training set that is both concise and representative. This core training set, used for large model training, can improve the model's generalization ability while reducing computational overhead.

Description

Technical Field

[0001] This application relates to the field of large model training technology, and in particular to a core training set construction method, a large model training method, and related apparatus. Background Technology

[0002] Currently, large-scale model technology has entered a stage of high-quality development. The final quality of the core training set (a key sample set selected from the original large-scale samples to achieve ideal generalization performance with low training cost) directly determines the generalization ability and training cost control level of the large model. Current core training set construction methods are mainly based on selection schemes with fixed sample size.

[0003] Among them, the screening scheme based on fixed sample size is to pre-set a fixed number of samples as the screening target, screen samples from the candidate sample set, and terminate the screening process when the number of screened samples reaches the preset number of samples, and the screened samples form the core training set.

[0004] While the above approach can construct a core training set, it suffers from the following problems due to its reliance on a pre-defined fixed number of samples: when the sample redundancy of the candidate sample set is high, a large number of duplicate samples are included in the core training set, causing redundant expansion of training data and increasing unnecessary computational overhead; when the sample diversity of the candidate sample set is high, a large number of unique semantic samples are not covered before the screening process is terminated, resulting in insufficient semantic coverage of the core training set and affecting the generalization ability of large models. Summary of the Invention

[0005] In view of this, this application provides a core training set construction method, a large model training method, and related apparatus to solve the technical problems of existing solutions, which rely on a preset fixed number of samples, resulting in redundant expansion of the core training set and increased computational overhead when the candidate sample set has high redundancy, and insufficient semantic coverage of the core training set when the candidate sample set has high diversity, thus affecting the generalization ability of the large model. The technical solution is as follows:

[0006] The first aspect of this application provides a core training set construction method, including:

[0007] Obtain a candidate sample set, which includes multiple samples;

[0008] An iterative screening method is used to select samples from the candidate sample set and add them to the initially empty first core training set until the iteration termination condition is met to obtain the final first core training set. The iteration termination condition is that after the samples are added in the most recent iteration, the semantic coverage gain of the first core training set is less than a preset coverage gain threshold. The semantic coverage gain represents the incremental value brought by the samples added in the most recent iteration to the first core training set in terms of semantic coverage compared to before the samples were added.

[0009] Based on the semantic distance between samples, semantically non-redundant samples are selected from the final first core training set to construct a second core training set, which is used to train the large model.

[0010] In one possible implementation, the iterative screening method for selecting samples from the candidate sample set to add to the initially empty first core training set includes:

[0011] Construct an initial empty first core training set;

[0012] One sample is selected from the candidate sample set and added to the first core training set;

[0013] If the current number of samples in the first core training set is less than the preset first sample number threshold, then return to the previous step to select one sample from the candidate sample set and add it to the first core training set.

[0014] If the current number of samples in the first core training set is greater than or equal to the first sample number threshold, then calculate the semantic coverage gain of the first core training set after adding samples this time.

[0015] If the semantic coverage gain is greater than or equal to the preset coverage gain threshold, then return to the step of selecting a sample from the candidate sample set and adding it to the first core training set;

[0016] If the semantic coverage gain is less than the coverage gain threshold, the iterative selection ends, and the final first core training set is obtained.

[0017] In one possible implementation, selecting a sample from the candidate sample set to add to the first core training set includes:

[0018] From the samples in the candidate sample set that have not yet been added to the first core training set, select the sample that can maximize the semantic coverage of the first core training set.

[0019] Add the selected samples to the first core training set.

[0020] In one possible implementation, calculating the semantic coverage gain of the first core training set after the addition of samples includes:

[0021] Calculate the coverage radius of the first core training set after the addition of samples to obtain the first coverage radius, where the first coverage radius characterizes the degree of dispersion of samples in the semantic space of the first core training set after the addition of samples.

[0022] Obtain the coverage radius of the first core training set before adding the sample, and then obtain the second coverage radius;

[0023] Calculate the difference between the first coverage radius and the second coverage radius, and use the calculated difference as the semantic coverage gain of the first core training set after adding samples.

[0024] In one possible implementation, calculating the coverage radius of the first core training set after the addition of samples includes:

[0025] For each sample in the first core training set after the addition of samples, calculate the semantic distance between the sample and each other sample in the first core training set after the addition of samples, and determine the minimum semantic distance among the calculated semantic distances as the coverage radius of the sample.

[0026] The maximum coverage radius among the coverage radii of all samples in the first core training set after the addition of samples is determined as the coverage radius of the first core training set after the addition of samples.

[0027] In one possible implementation, the step of selecting semantically non-redundant samples from the final first core training set to construct a second core training set based on the semantic distance between samples includes:

[0028] Construct an initially empty second core training set;

[0029] Seed samples are selected from the final first core training set according to a preset screening strategy and added to the second core training set;

[0030] Iterate through all samples in the final first core training set except for the seed samples, and perform the following operations on the currently iterated sample:

[0031] Calculate the semantic distance between the currently traversed sample and each sample in the second core training set. Based on the calculated semantic distances, determine whether the currently traversed sample is a redundant sample. If the currently traversed sample is determined to be a non-redundant sample, then add the currently traversed sample to the second core training set.

[0032] After the traversal is completed, the final second core training set is obtained.

[0033] In one possible implementation, determining whether the currently traversed sample is a redundant sample based on the calculated semantic distances includes:

[0034] The number of samples in the second core training set whose semantic distance from the currently traversed sample is less than a preset redundancy distance threshold is counted.

[0035] If the number of samples obtained from the statistics is greater than or equal to the preset second sample number threshold, then the sample currently being traversed is determined to be a redundant sample.

[0036] If the number of samples obtained from the statistics is less than the second sample number threshold, then the sample currently being traversed is determined to be a non-redundant sample.

[0037] In one possible implementation, the step of selecting seed samples from the final first core training set and adding them to the second core training set according to a preset screening strategy includes:

[0038] Determine the comprehensive score of each sample in the final first core training set across multiple evaluation dimensions;

[0039] The sample with the highest overall score in the final first core training set is used as the seed sample and added to the second core training set.

[0040] In one possible implementation, calculating the semantic distance between the currently traversed sample and each sample in the second core training set includes:

[0041] Based on the semantic embedding vector of the currently traversed sample and the semantic embedding vector of each sample in the second core training set, calculate the semantic distance between the currently traversed sample and each sample in the second core training set.

[0042] The core training set construction method also includes:

[0043] After adding the seed sample to the second core training set, a semantic embedding vector index list for the second core training set is established. The initial semantic embedding vector index list contains the semantic embedding vector index corresponding to the seed sample.

[0044] If the currently traversed sample is determined to be a non-redundant sample and is added to the second core training set, the semantic embedding vector index corresponding to the currently traversed sample is added to the semantic embedding vector index list so that the semantic embedding vector of each sample in the second core training set can be located through the semantic embedding vector index list when calculating the semantic distance later.

[0045] The second aspect of this application provides a method for training large models, including:

[0046] Obtain the target core training set, which is a second core training set constructed using any of the above-mentioned core training set construction methods;

[0047] The large model is trained using the target core training set.

[0048] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0049] The memory is used to store computer programs;

[0050] The processor is used to execute the computer program to enable the electronic device to implement any of the above-described core training set construction methods, or to implement the above-described large model training methods.

[0051] A fourth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement any of the aforementioned core training set construction methods or the aforementioned large model training methods.

[0052] The fifth aspect of this application provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the above-described core training set construction methods, or to implement the above-described large model training methods.

[0053] Using the above technical solution, the core training set construction method provided in this application first constructs a first core training set through iterative screening, and then adopts an iterative termination mechanism based on semantic coverage gain. This allows the screening process to automatically determine the optimal termination point according to the semantic distribution of the candidate sample set. Specifically, when the redundancy of the candidate sample set is high, the marginal contribution of newly added samples decreases rapidly, and the iteration terminates early, avoiding the expansion of duplicate samples. When the diversity of the candidate sample set is high, multiple iterations are required before the semantic coverage gain drops below the coverage gain threshold, ensuring that unique semantic samples are fully covered. The above iterative screening achieves a dynamic balance between screening efficiency and the quality of the first core training set. Considering that there are still redundant samples in the first core training set, especially implicit redundant samples, this application further introduces a second-stage screening. The second-stage screening aims to eliminate redundant samples and achieve deep purification of samples. The two stages work together. The first stage covers diverse semantic regions through iterative screening to ensure semantic breadth, while the second stage eliminates redundant samples through semantic distance screening to achieve deep purification. The two stages take into account both the representativeness and conciseness of the core training set. The core training set construction method provided in this application can construct a second core training set that is concise, representative, and highly valuable. This second core training set is used for training large models. On the one hand, it can effectively reduce storage space, computing power consumption, and training time. On the other hand, it can improve the instruction compliance ability and task generalization ability of large models. That is, the second core training set constructed in this application can improve the generalization ability of models while reducing computational overhead, helping large models to upgrade from "usable" to "easy to use", which has important practical application value. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 A flowchart illustrating the core training set construction method provided in the embodiments of this application;

[0056] Figure 2 The flowchart provided in this application embodiment illustrates the process of selecting samples from the candidate sample set and adding them to the initially empty first core training set using an iterative screening method until the iteration termination condition is met, thus obtaining the final first core training set.

[0057] Figure 3This is a flowchart illustrating the process of constructing a second core training set by selecting semantically non-redundant samples from the final first core training set based on the semantic distance between samples, as provided in the embodiments of this application.

[0058] Figure 4 This is a flowchart illustrating the process of determining whether a currently traversed sample is a redundant sample based on the calculated semantic distance, as provided in an embodiment of this application.

[0059] Figure 5 This is a schematic diagram of the core training set construction device provided in the embodiments of this application. Detailed Implementation

[0060] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0061] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0062] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0063] The screening scheme based on fixed sample size relies on a preset fixed number of samples, which leads to the redundancy expansion of the core training set and increased computational overhead when the candidate sample set has high redundancy. At the same time, when the candidate sample set has high diversity, the semantic coverage of the core training set is insufficient, affecting the generalization ability of the large model.

[0064] In response to the problems of existing solutions, the inventors of this case conducted research. The initial idea was to use a screening scheme based on static similarity thresholds to screen samples and construct a core training set. That is, a fixed semantic similarity threshold is set in advance, and samples in the candidate sample set are traversed. Only when the semantic similarity between the traversed sample and each sample already selected in the core training set is lower than the threshold is the sample added to the core training set.

[0065] The inventors in this case, through research on screening schemes based on static similarity thresholds, discovered that when the similarity threshold is set too strictly, the sample inclusion threshold is too high, the core training set is too small, and it is difficult to cover the main regions of the semantic space, resulting in insufficient representativeness of the core training set. When the similarity threshold is set too loosely, a large number of redundant samples are included in the core training set, leading to redundancy expansion of the core training set and increasing computational overhead. In addition, screening schemes based on static similarity thresholds only rely on pairwise similarity comparisons for deduplication, without considering the global semantic relationships between samples within the core training set. This can easily lead to the accidental deletion of semantically unique high-value samples or the omission of implicit semantic redundancy data.

[0066] To address the problems of the aforementioned solutions, the inventors of this case, through continuous research, finally proposed a more effective core training set construction method. This method balances simplicity and representativeness, effectively controlling the scale of training data while ensuring the integrity of semantic coverage, thus providing a high-quality core training set for large models.

[0067] The core training set construction method provided in this application will be described below through the following embodiments.

[0068] Please see Figure 1 The diagram illustrates a flowchart of the core training set construction method provided in this application embodiment. This core training set construction method may include:

[0069] Step S101: Obtain the candidate sample set.

[0070] The candidate sample set includes a large number of samples.

[0071] Step S102: Using an iterative screening method, samples are selected from the candidate sample set and added to the initially empty first core training set until the iteration termination condition is met, thus obtaining the final first core training set.

[0072] The iteration termination condition is that after the samples are added in the most recent iteration, the semantic coverage gain of the first core training set is less than the preset coverage gain threshold.

[0073] It should be noted that the semantic coverage gain mentioned above represents the incremental value brought by the samples added in the most recent iteration to the semantic coverage of the first core training set compared to before the samples were added. In other words, the samples added in the most recent iteration represent the incremental value of semantic coverage brought by the first core training set.

[0074] This step aims to select a first core training set with broad semantic coverage from a candidate sample set containing a large number of samples, that is, a first core training set with broad semantic representation.

[0075] Step S103: Based on the semantic distance between samples, select semantically non-redundant samples from the final first core training set to construct the second core training set.

[0076] This step involves a second round of screening based on the first core training set. The purpose is to deeply refine the first core training set, which already has broad semantic representativeness, remove semantically redundant samples, and construct a more compact and efficient second core training set.

[0077] Specifically, although the first core training set ensures the breadth of semantic coverage through iterative selection, it may still contain redundant samples with high semantic similarity, especially some implicit redundant samples (such as samples with different expressions but the same semantic essence). If these redundant samples are directly used for training large models, it will not only increase the computational cost, but may also cause the model to overfit to specific expressions, affecting the generalization ability.

[0078] Therefore, in this embodiment, after obtaining the final first core training set, the semantic distance between samples is used as the selection criterion to select samples from the final first core training set that meet the preset semantic non-redundancy condition to construct the second core training set. The semantic distance between two samples is determined based on their semantic embedding vectors. For example, the cosine distance between the semantic embedding vectors of two samples can be calculated as the semantic distance between them. The semantic distance between two samples measures the degree of semantic similarity between them; the smaller the semantic distance, the more similar the two samples are semantically, and the larger the semantic distance, the greater the semantic difference between the two samples.

[0079] The core training set construction method provided in this application can be used to construct the core training set of a general-domain large model. The second core training set (including labeled training samples from the general domain) constructed using this method for the general-domain large model has the characteristics of broad semantic coverage and low redundancy. Training the general-domain large model with this second core training set can effectively reduce the storage space, computing power consumption, and training time required for training the model. Simultaneously, it can improve the instruction compliance and task generalization capabilities of the general-domain large model, helping it upgrade from "usable" to "easy to use," and has significant practical application value.

[0080] The core training set construction method provided in this application embodiment can also be used to construct the core training set of a large vertical domain model. The second core training set (containing labeled training samples from the vertical domain) constructed using the core training set construction method provided in this application embodiment for a large vertical domain model has the characteristics of comprehensive domain semantic coverage and low redundancy. Using this second core training set to train the large vertical domain model can, on the one hand, achieve efficient coverage of the domain semantic space with limited sample capacity, reducing the storage cost of domain data and training computation overhead; on the other hand, it can enhance the model's understanding of domain terminology, domain logic, and domain tasks, improving the model's instruction compliance accuracy and task execution performance in vertical scenarios, thus helping the large vertical domain model achieve low-cost, high-efficiency domain adaptation and performance optimization.

[0081] For example, the second core training set constructed by the core training set construction method provided in the embodiments of this application can be a core training set used to train a large model in the contract domain (such as a large model for contract review). It can include contract-related data samples in the contract domain, such as contract clause samples labeled with the type of "liability for breach of contract" such as "If Party A fails to make payment on time, it shall pay a penalty of 0.05% of the unpaid amount for each day of delay", contract text samples labeled with the level of "high risk" such as "This contract does not stipulate confidentiality obligations and liability for breach of contract", contract quality inspection samples labeled with "missing confidentiality clauses", and key element extraction samples of contracts labeled with "contract amount of RMB 1 million and contract term of 2 years", etc.

[0082] For example, the second core training set constructed by the core training set construction method provided in the embodiments of this application can be a core training set used to train large models in the financial field. It can include financial-related data samples in the financial field, such as financial news samples labeled with "negative" sentiment, such as "the central bank announced an interest rate hike of 25 basis points", financial product review samples labeled with "violation", and financial report paragraphs of listed companies labeled with "high risk level".

[0083] For example, the second core training set constructed by the core training set construction method provided in the embodiments of this application can be a core training set for training a large model in the field of bidding and tendering. It may include bidding and tendering related data samples in the field of bidding and tendering, such as bidding announcement samples labeled with the project type "smart transportation", winning bid announcement samples labeled with "winning bid discount rate of 5.36%", and bid review samples labeled with "reason for rejection: qualification mismatch", etc.

[0084] For example, the second core training set constructed by the core training set construction method provided in the embodiments of this application can be a core training set for training large models in the supply chain domain. It can include material-related data samples in the supply chain domain, such as material description samples labeled with the category of "fastener-bolt", material code samples labeled with the standard code GB / T 5782, and material pair samples labeled with the relationship of "synonyms" such as "external hexagonal bolt" and "hexagonal head bolt".

[0085] For example, the second core training set constructed by the core training set construction method provided in the embodiments of this application can be a core training set for training large models in the field of intelligent customer service and marketing. It can include interaction-related data samples in the field of intelligent customer service and marketing, such as user question samples labeled with the intent of "checking bills" such as "How much did I spend on my credit card this month?", customer service dialogue samples labeled with the emotion of "anger", and outbound call transcription text samples labeled with the "high intent" tag.

[0086] For example, the second core training set constructed by the core training set construction method provided in the embodiments of this application can be a core training set used to train a large model in the field of financial transactions. It may include transaction-related data samples in the field of financial transactions, such as user consumption record samples labeled with "fresh food preference", credit card transaction flow samples labeled with "high risk level", supermarket card swiping data samples labeled with "merchant category MCC 5411", etc.

[0087] The core training set construction method provided in this application first constructs a first core training set using an iterative screening approach and employs an iterative termination mechanism based on semantic coverage gain. This allows the screening process to automatically determine the optimal termination point based on the semantic distribution of the candidate sample set. Specifically, when the candidate sample set has high redundancy, the marginal contribution of newly added samples decreases rapidly, and the iteration terminates early to avoid the expansion of duplicate samples and solve the problem of increased computational overhead. When the candidate sample set has high diversity, multiple rounds of iteration are required before the semantic coverage gain drops below the coverage gain threshold, ensuring sufficient coverage of unique semantic samples and solving the problem of insufficient semantic coverage affecting the model's generalization ability. The first-stage screening achieves a dynamic balance between screening efficiency and core training set quality. Considering that there are still redundant samples in the first core training set, especially implicit redundant samples, this application further introduces a second-stage screening. The second-stage screening aims to deeply purify the first core training set, which already has broad semantic representativeness, by removing semantically redundant samples. The two stages work together: the first stage covers diverse semantic regions through iterative screening to ensure semantic breadth, and the second stage eliminates redundant samples through semantic distance screening to achieve deep purification. The two stages balance the representativeness and conciseness of the core training set.

[0088] The core training set construction method provided in this application embodiment can construct a second core training set that is concise, representative, and highly valuable. This second core training set is used for training large models. On the one hand, it can effectively reduce storage space, computing power consumption, and training time. On the other hand, it can improve the instruction compliance ability and task generalization ability of large models. That is, the second core training set constructed through this application embodiment can improve the model's generalization ability while reducing computational overhead, helping large models upgrade from "usable" to "easy to use", and has important practical application value.

[0089] In some embodiments of this application, the implementation process of "step S102: using an iterative screening method to select samples from the candidate sample set and add them to the initially empty first core training set until the iteration termination condition is met, and the final first core training set is obtained" in the above embodiment is described.

[0090] like Figure 2 As shown, the process of selecting samples from the candidate sample set and adding them to the initially empty first core training set using an iterative screening method until the iteration termination condition is met, to obtain the final first core training set, may include:

[0091] Step S201: Construct the first core training set, which is initially empty.

[0092] This step is the initialization operation. An empty first core training set is created. At this time, the first core training set has no samples and the semantic coverage is zero. This empty set serves as the starting point for subsequent iterations. In each subsequent iteration, samples will be gradually selected from the candidate sample set and added to this set.

[0093] Step S202: Select samples from the candidate sample set and add them to the first core training set.

[0094] In one possible implementation, samples that can maximize the semantic coverage of the first core training set can be selected from the samples in the candidate sample set that have not yet been added to the first core training set, and the selected samples can be added to the first core training set.

[0095] Iterate through the candidate sample set for samples that have not yet been added to the first core training set. For the sample currently being iterated, evaluate how much the semantic coverage of the first core training set can be expanded if it is added to the first core training set. Then select the sample that can bring the greatest expansion effect and officially add it to the first core training set.

[0096] In one possible implementation, for each sample in the candidate sample set that has not yet been added to the first core training set, the coverage radius of the first core training set after adding the sample can be calculated as the coverage radius corresponding to the sample. The maximum coverage radius is determined from the coverage radii corresponding to each sample in the candidate sample set that has not yet been added to the first core training set. The sample corresponding to the maximum coverage radius is determined as the sample that can maximize the semantic coverage of the first core training set, and then it is added to the first core training set.

[0097] Step S203: Determine whether the current number of samples in the first core training set is greater than or equal to the preset first sample number threshold.

[0098] If the current number of samples in the first core training set is less than the preset first sample number threshold (e.g., 2), then return to step S202; if the current number of samples in the first core training set is greater than or equal to the first sample number threshold, then execute step S204 and subsequent steps.

[0099] If the current number of samples in the first core training set is less than the preset first sample number threshold (e.g., 2), it means that the size of the first core training set is too small. Instead of terminating the iteration, the iteration continues directly, that is, samples are selected from the candidate sample set and added to the first core training set. This process is repeated until the size of the first core training set reaches the first sample number threshold.

[0100] If the current number of samples in the first core training set is greater than or equal to the first sample number threshold (e.g., 2), it means that the first core training set has a sufficient basic size. At this time, the algorithm will determine whether to terminate the iteration, that is, to enter steps S204 and S205, that is, to calculate the semantic coverage gain brought by the added samples and determine whether this semantic coverage gain is less than the preset coverage gain threshold.

[0101] In this embodiment, the first sample quantity threshold is essentially a switch. Before the size of the first core training set reaches this threshold, only one thing is done: continuously selecting samples from the candidate sample set to add to the first core training set, without considering whether to stop. Once the size of the first core training set exceeds this threshold, an evaluation mechanism is activated. That is, the semantic coverage gain is calculated once for each added sample, and the iteration is determined based on whether the semantic coverage gain falls below the coverage gain threshold. This design ensures the basic representativeness of the first core training set and avoids making incorrect termination judgments when the size is too small.

[0102] Step S204: Calculate the semantic coverage gain of the first core training set after adding the samples.

[0103] The semantic coverage gain representation of the first core training set after the addition of samples is the incremental value brought to the semantic coverage of the first core training set after the addition of samples compared to before the addition of samples. In other words, the incremental value of semantic coverage brought to the first core training set by the added samples.

[0104] In one possible implementation, the process of calculating the semantic coverage gain of the first core training set after the addition of samples may include:

[0105] Step S2041: Calculate the coverage radius of the first core training set after the addition of samples to obtain the first coverage radius, and obtain the coverage radius of the first core training set before the addition of samples to obtain the second coverage radius.

[0106] The first coverage radius represents the degree of dispersion of samples in the first core training set in the semantic space after the addition of samples, while the second coverage radius represents the degree of dispersion of samples in the first core training set in the semantic space before the addition of samples.

[0107] For example, if the most recent iteration is the 5th iteration, then the first coverage radius is the coverage radius of the first core training set after the 5th iteration, and the second coverage radius is the coverage radius of the first core training set after the 4th iteration.

[0108] In one possible implementation, the coverage radius of the first core training set after the addition of samples is called the first coverage radius. It can be calculated using the following formula:

[0109] (1).

[0110] Where m represents the number of samples in the first core training set after the addition of samples in this iteration (i.e., the first core training set after the most recent iteration of adding samples). This represents the semantic distance between the i-th sample and the j-th sample in the first core training set after the addition of this sample. Let represent the semantic embedding vector of the i-th sample and the semantic embedding vector of the j-th sample.

[0111] Optional, It can be calculated using the following formula:

[0112] (2).

[0113] in, for The length of the module.

[0114] Equation (1) above means that for each sample in the first core training set after the addition of samples this time (i.e., the first core training set after the most recent iteration of adding samples), the semantic distance between the sample and each other sample in the first core training set after the addition of samples this time is calculated, and the minimum semantic distance among the calculated semantic distances is determined as the coverage radius of the sample, and the maximum coverage radius among the coverage radii of each sample in the first core training set after the addition of samples this time is determined as the coverage radius of the first core training set after the addition of samples this time.

[0115] It should be noted that the calculation method for the second coverage radius is the same as that for the first coverage radius.

[0116] Step S2042: Calculate the difference between the first coverage radius and the second coverage radius, and use the calculated difference as the semantic coverage gain of the first core training set after adding samples this time.

[0117] If we represent the semantic coverage gain of the first core training set after adding samples as... ,but for:

[0118] (3).

[0119] in, This represents the second coverage radius, which is the coverage radius of the first core training set before the current sample was added.

[0120] By quantifying the semantic coverage gain, an objective basis for termination judgment is provided, which improves the stability and reproducibility of the scheme under different datasets.

[0121] Step S205: Determine whether the semantic coverage gain of the first core training set after adding samples is less than the preset coverage gain threshold.

[0122] If the semantic coverage gain of the first core training set after adding samples is greater than or equal to the preset coverage gain threshold (e.g., 0.05), then return to step S202; if the semantic coverage gain of the first core training set after adding samples is less than the coverage gain threshold, then end the iterative selection and obtain the final first core training set.

[0123] If the semantic coverage gain of the first core training set after adding samples is greater than or equal to the preset coverage gain threshold, it means that the added samples have significantly expanded the semantic coverage of the first core training set. The first core training set is not yet saturated and will continue to iterate.

[0124] If the semantic coverage gain of the first core training set after adding samples is less than the preset coverage gain threshold, it means that the marginal contribution of the added samples is very limited and the first core training set is close to saturation. At this time, the entire iterative selection process will be terminated, thus obtaining the final first core training set.

[0125] The iterative termination mechanism based on semantic coverage gain enables the screening process to automatically determine the optimal termination point according to the semantic distribution of the candidate sample set, without the need for manual preset of the number of samples, thus avoiding undercoverage and overcoverage problems.

[0126] The process of constructing the first core training set will be explained below with a specific example.

[0127] Suppose we need to select samples from a candidate sample set containing a large number of financial text samples to construct a core training set for training a large financial model. The candidate sample set contains the following 10 financial text samples, covering different topics such as financial statement analysis, market conditions, risk control, and investment strategies. Some samples have semantic overlap:

[0128] Sample 1: The company's net profit increased by 15% year-on-year, mainly due to the growth in main business revenue and the optimization of cost control.

[0129] Sample 2: The company's net profit increased by 16% year-on-year, mainly due to the increase in main business revenue and the decrease in management expenses.

[0130] Sample 3: A company's quarterly revenue reached 10 billion yuan, exceeding market expectations, and its stock price rose 5% in after-hours trading.

[0131] Sample 4: A company's quarterly revenue was 9.8 billion yuan, slightly higher than analysts' expectations, and its stock price rose slightly.

[0132] Sample 5: With increased market volatility, investors should focus on diversifying their asset allocation to mitigate risk.

[0133] Sample 6: Given the current high level of market uncertainty, it is recommended to reduce portfolio volatility risk through diversified investment.

[0134] Sample 7: In credit risk assessment, the default probability model can be used to make predictions by combining corporate financial indicators and macroeconomic factors.

[0135] Sample 8: Credit rating downgrades may lead to increased corporate financing costs, and attention should be paid to debt repayment capacity.

[0136] Sample 9: In quantitative investment strategies, multi-factor models can be used for stock selection and risk control.

[0137] Sample 10: Technical analysis indicators such as moving averages and relative strength index (RSI) can help determine market trends.

[0138] The preset parameters are as follows: the coverage gain threshold is set to 0.05, the first sample number threshold is set to 2, and each sample has been converted into a semantic embedding vector through a pre-trained language model.

[0139] The process of constructing the first core training set is as follows:

[0140] ① Construct an initial empty first core training set.

[0141] Create an empty set Set_Core 1 = {}, at which point there are no samples in the first core training set.

[0142] ② Conduct the first round of iterations.

[0143] Since the first core training set Set_Core 1 is empty, the first sample is selected from the candidate sample set. Let's say we select sample 1 and add it to Set_Core 1. At this time, Set_Core 1 = {sample 1}. Since the current number of samples in Set_Core 1 is less than the first sample number threshold of 2, we continue the iteration directly.

[0144] ③ Conduct the second round of iterations.

[0145] Samples are selected from the candidate sample set that have not yet been added to Set_Core 1 (including samples 2, 3, 4, 5, 6, 7, 8, 9, and 10). Assuming that sample 3 can maximize the semantic coverage of Set_Core 1, sample 3 is added to Set_Core 1. At this point, Set_Core 1 = {sample 1, sample 3}. Since the current number of samples in Set_Core 1 is equal to the first sample number threshold of 2, the semantic coverage gain of the first core training set after this addition is calculated. Assuming the calculated semantic coverage gain is 0.38, and since 0.38 is greater than the preset coverage gain threshold of 0.05, the iteration continues.

[0146] ④ Conduct the third round of iteration.

[0147] Samples are selected from the candidate sample set that have not yet been added to Set_Core 1 (including samples 2, 4, 5, 6, 7, 8, 9, and 10). Assuming sample 5 can maximize the semantic coverage of Set_Core 1, it is added to Set_Core 1, at which point Set_Core 1 = {sample 1, sample 3, sample 5}. Since the current number of samples in Set_Core 1 is greater than the first sample number threshold of 2, the semantic coverage gain of the first core training set after this addition is calculated. Assuming the calculated semantic coverage gain is 0.35, and since 0.35 is greater than the coverage gain threshold of 0.05, the iteration continues.

[0148] ⑤ Conduct the fourth iteration.

[0149] Samples are selected from the candidate sample set that have not yet been added to Set_Core 1 (including samples 2, 4, 6, 7, 8, 9, and 10). Assuming sample 7 can maximize the semantic coverage of Set_Core 1, it is added to Set_Core 1. At this point, Set_Core 1 = {sample 1, sample 3, sample 5, sample 7}. Since the current number of samples in Set_Core 1 is greater than the first sample number threshold of 2, the semantic coverage gain of the first core training set after this addition is calculated. Assuming the calculated semantic coverage gain is 0.30, and since 0.30 is greater than the coverage gain threshold of 0.05, the iteration continues.

[0150] ⑥ Conduct the fifth iteration.

[0151] Samples are selected from the candidate sample set that have not yet been added to Set_Core 1 (including samples 2, 4, 5, 6, 8, 9, and 10). Assuming sample 9 can maximize the semantic coverage of Set_Core 1, it is added to Set_Core 1. At this point, Set_Core 1 = {sample 1, sample 3, sample 5, sample 7, sample 9}. Since the current number of samples in Set_Core 1 is greater than the first sample number threshold of 2, the semantic coverage gain of the first core training set after this addition is calculated. Assuming the calculated semantic coverage gain is 0.28, and since 0.28 is greater than the coverage gain threshold of 0.05, the iteration continues.

[0152] ⑦ Perform the sixth iteration.

[0153] Samples are selected from the candidate sample set that have not yet been added to Set_Core 1 (including samples 2, 4, 6, 8, and 10). Assuming sample 10 can maximize the semantic coverage of Set_Core 1, it is added to Set_Core 1. At this point, Set_Core 1 = {sample 1, sample 3, sample 5, sample 7, sample 9, sample 10}. Since the current number of samples in Set_Core 1 is greater than the first sample number threshold of 2, the semantic coverage gain of the first core training set after this addition is calculated. Assuming the calculated semantic coverage gain is 0.15, and since 0.15 is greater than the coverage gain threshold of 0.05, the iteration continues.

[0154] ⑧ Conduct the seventh iteration.

[0155] Samples are selected from the candidate sample set that have not yet been added to Set_Core 1 (including samples 2, 4, 6, and 8). Assuming sample 8 can maximize the semantic coverage of Set_Core 1, it is added to Set_Core 1. At this point, Set_Core 1 = {sample 1, sample 3, sample 5, sample 7, sample 9, sample 10, sample 8}. Since the current number of samples in Set_Core 1 is greater than the first sample number threshold of 2, the semantic coverage gain of Set_Core 1 after this iteration is calculated. Assuming the calculated semantic coverage gain is 0.09, and since 0.09 is greater than the coverage gain threshold of 0.05, the iteration continues.

[0156] ⑨ Proceed to the eighth iteration.

[0157] Samples are selected from the candidate sample set that have not yet been added to Set_Core 1 (including samples 2, 4, and 6). Assuming sample 4 can maximize the semantic coverage of Set_Core 1, it is added to Set_Core 1. At this point, Set_Core 1 = {sample 1, sample 3, sample 5, sample 7, sample 9, sample 10, sample 8, sample 4}. Since the current number of samples in Set_Core 1 is greater than the first sample number threshold of 2, the semantic coverage gain of the first core training set after this sample addition is calculated. Assuming the calculated semantic coverage gain is 0.04, and since 0.04 is less than the coverage gain threshold of 0.05, the iteration ends.

[0158] ⑩ After the iteration is completed, the final first core training set Set_Core 1 is obtained.

[0159] The final first core training set, Set_Core 1, can be either Set_Core 1 from the last iteration or Set_Core 1 from the iteration before the last iteration.

[0160] The final selected samples cover seven different financial semantic areas, including company financial growth analysis, company revenue and market response, market risk and asset allocation, credit risk assessment, quantitative investment strategies, technical analysis indicators, and the impact of credit rating, covering the main business scenarios in the financial field.

[0161] After obtaining the final first core training set, considering that there may be redundant samples, especially implicit redundant samples, in the first core training set, the second core training set is constructed by further selecting samples that meet the preset semantic non-redundancy conditions from the final first core training set based on the semantic distance between samples. This process is described in some embodiments of this application.

[0162] like Figure 3 As shown, the process of selecting semantically non-redundant samples from the final first core training set to construct the second core training set, based on the semantic distance between samples, may include:

[0163] Step S301: Construct an initially empty second core training set.

[0164] This step is an initialization operation. An empty second core training set is created, meaning that the second core training set currently contains no samples.

[0165] Step S302: Select seed samples from the final first core training set and add them to the second core training set according to the preset screening strategy.

[0166] In one possible implementation, the comprehensive score of each sample in the final first core training set across multiple evaluation dimensions can be determined, and then the sample with the highest comprehensive score in the final first core training set can be added as a seed sample to the initially empty second core training set.

[0167] For each sample in the final first core training set, the sample can be evaluated separately across multiple evaluation dimensions to obtain a score for each dimension. These scores are then combined (e.g., by direct summation, weighted summation, etc.) to obtain a comprehensive score for the sample across multiple evaluation dimensions. These multiple evaluation dimensions may include, but are not limited to, some or all of the following: semantic representativeness, data completeness, information novelty, semantic diversity, etc.

[0168] It should be noted that the seed sample is the starting point of the entire screening process and the first sample to be included in the second core training set. Selecting a high-quality seed sample is crucial. This embodiment adopts a screening strategy based on the highest comprehensive score. The sample with the highest comprehensive score is the best or most important sample, and using it as the seed sample ensures that the screening process starts with a sample that has the most information and the highest quality.

[0169] Optionally, after adding the seed sample to the second core training set, a semantic embedding vector index list for the second core training set can be established. The initial semantic embedding vector index list contains the semantic embedding vector index corresponding to the seed sample (the index of the semantic embedding vector of the seed sample). For example, the initial semantic embedding vector index list I=[0], where "0" is the semantic embedding vector index corresponding to the seed sample.

[0170] Step S303: Traverse all samples in the final first core training set except for the seed sample, and perform the following for the currently traversed sample:

[0171] Step S3031: Calculate the semantic distance between the currently traversed sample and each sample in the second core training set.

[0172] The semantic distance between two samples measures the degree of semantic similarity between them.

[0173] Specifically, the semantic distance between the currently traversed sample and each sample in the second core training set can be calculated based on the semantic embedding vector of the currently traversed sample and the semantic embedding vector of each sample in the second core training set. Optionally, the cosine distance between the semantic embedding vector of the currently traversed sample and the semantic embedding vector of each sample in the second core training set can be calculated as the semantic distance. The calculation method of the cosine distance is as shown in Equation (2).

[0174] Step S3032: Based on the calculated semantic distance, determine whether the currently traversed sample is a redundant sample.

[0175] like Figure 4 As shown, the process of determining whether a currently traversed sample is a redundant sample based on the calculated semantic distance can include:

[0176] Step S3032-1: Count the number of samples in the second core training set whose semantic distance from the currently traversed sample is less than a preset distance threshold (e.g., 0.3).

[0177] This step aims to determine how many samples in the second core training set are semantically very similar to the currently traversed samples.

[0178] Step S3032-2: Determine whether the number of samples obtained from the statistics is less than the preset second sample number threshold.

[0179] If the number of samples obtained is less than the preset second sample number threshold, then step S3032-3a is executed; if the number of samples obtained is greater than or equal to the preset second sample number threshold (e.g., 2), then step S3032-3b is executed.

[0180] Step S3032-3a: Determine that the currently traversed sample is a non-redundant sample.

[0181] Step S3032-3b: Determine that the currently traversed sample is a redundant sample.

[0182] After obtaining the number of samples, it is determined whether the number of samples obtained is less than a preset second sample number threshold. If the number of samples obtained is less than the second sample number threshold, it means that there are almost no samples in the second core training set that are semantically very similar to the currently traversed sample. That is, the currently traversed sample brings new and non-repetitive information. Therefore, the currently traversed sample is determined to be a non-redundant sample. If the number of samples obtained is greater than or equal to the preset second sample number threshold, it means that there are already enough samples in the second core training set that are semantically very similar to the currently traversed sample. At this time, the information that the currently traversed sample can provide has been fully reflected in the second core training set. Therefore, the currently traversed sample is determined to be a redundant sample.

[0183] If the sample being iterated is a non-redundant sample, then proceed to step S3033.

[0184] Step S3033: Add the currently traversed sample to the second core training set.

[0185] After adding the currently traversed sample to the second core training set, the semantic embedding vector index corresponding to the currently traversed sample can be added to the semantic embedding vector index list, so that the semantic embedding vector of each sample in the second core training set can be located by the semantic embedding vector index list when calculating the semantic distance later.

[0186] After the traversal is completed, the final second core training set is obtained.

[0187] The second core training set, constructed through semantic redundancy removal, is a refined sample set with high diversity and low redundancy. It can cover more comprehensive semantic information with fewer samples, thus significantly improving training efficiency, generalization ability, and interpretability when used to train large models.

[0188] The following example will illustrate the process of constructing the second core training set.

[0189] Suppose the final first core training set Set_Core1 = {sample 1, sample 3, sample 5, sample 7, sample 9, sample 10, sample 8}, where:

[0190] Sample 1: The company's net profit increased by 15% year-on-year, mainly due to the growth in main business revenue and the optimization of cost control.

[0191] Sample 3: A company's quarterly revenue reached 10 billion yuan, exceeding market expectations, and its stock price rose 5% in after-hours trading.

[0192] Sample 5: With increased market volatility, investors should focus on diversifying their asset allocation to mitigate risk.

[0193] Sample 7: In credit risk assessment, the default probability model can be used to make predictions by combining corporate financial indicators and macroeconomic factors.

[0194] Sample 8: Credit rating downgrades may lead to increased corporate financing costs, and attention should be paid to debt repayment capacity.

[0195] Sample 9: In quantitative investment strategies, multi-factor models can be used for stock selection and risk control.

[0196] Sample 10: Technical analysis indicators such as moving averages and relative strength index (RSI) can help determine market trends.

[0197] The preset parameters are as follows: the redundancy distance threshold is set to 0.25, and the second sample number threshold is set to 1.

[0198] The process of constructing the second core training set is as follows:

[0199] ① Construct a second core training set that is initially empty.

[0200] Create an empty set Set_Core 2 = {}. At this point, the second core training set Set_Core 2 contains no samples.

[0201] ② Select seed samples from the first core training set Set_Core 1 and add them to the second core training set Set_Core2.

[0202] Determine the overall score of each sample in Set_Core 1 across multiple dimensions. Assume that the overall score of sample 1 is 96, the overall score of sample 3 is 94, the overall score of sample 5 is 91, the overall score of sample 7 is 93, the overall score of sample 8 is 88, the overall score of sample 9 is 89, and the overall score of sample 10 is 86.

[0203] From the 7 samples in Set_Core 1, the sample with the highest overall score is selected as the seed sample. Sample 1 has the highest overall score, so it is added to the second core training set Set_Core 2 as the seed sample. At this point, Set_Core 2 = {sample 1}. Simultaneously, a semantic embedding vector index list is created, initially containing the semantic embedding vector index of sample 1.

[0204] ③ Traverse the remaining 6 samples in Set_Core 1, excluding sample 1:

[0205] Traversing to sample 3:

[0206] First, the semantic distance between sample 3 and each sample in Set_Core 2 is calculated. Set_Core 2 currently only contains sample 1. The semantic distance between sample 3 and sample 1 is calculated to be 0.35. Then, it is determined whether sample 3 is a redundant sample. Specifically, the number of samples in Set_Core 2 whose semantic distance to sample 3 is less than the redundancy distance threshold of 0.25 is counted. The distance between sample 3 and sample 1 is 0.35, which is greater than 0.25. Therefore, the counted sample number is 0. Since 0 is less than the second sample number threshold of 1, sample 3 is determined to be a non-redundant sample. Then, sample 3 is added to Set_Core 2. At this time, Set_Core 2 = {sample 1, sample 3}.

[0207] Traversing to sample 5:

[0208] The semantic distance between sample 5 and each sample in Set_Core 2 is calculated. The semantic distance between sample 5 and sample 1 is 0.42, and the semantic distance between sample 5 and sample 3 is 0.38. Since both 0.42 and 0.38 are greater than 0.25, the number of samples in Set_Core 2 with a semantic distance less than the redundancy distance threshold of 0.25 is 0. Since the number of samples obtained is less than the second sample number threshold of 1, sample 5 is determined to be a non-redundant sample. Therefore, sample 5 is added to Set_Core 2, and Set_Core 2 at this time = {sample 1, sample 3, sample 5}.

[0209] Traversing to sample 7:

[0210] Calculate the semantic distance between sample 7 and each sample in Set_Core 2. The semantic distance between sample 7 and sample 1 is 0.28, the semantic distance between sample 7 and sample 3 is 0.44, and the semantic distance between sample 7 and sample 5 is 0.39. Count the number of samples in Set_Core 2 whose semantic distance to sample 7 is less than the redundancy distance threshold of 0.25. The result is 0, which is less than the second sample number threshold of 1. Therefore, sample 7 is determined to be a non-redundant sample, and sample 7 is added to Set_Core 2. At this time, Set_Core 2 = {sample 1, sample 3, sample 5, sample 7}.

[0211] Traversing to sample 8:

[0212] The semantic distance between sample 8 and each sample in Set_Core 2 is calculated. The semantic distance between sample 8 and sample 1 is 0.31, between sample 8 and sample 3 is 0.46, between sample 8 and sample 5 is 0.42, and between sample 8 and sample 7 is 0.12. The number of samples in Set_Core 2 whose semantic distance to sample 8 is less than the redundancy distance threshold of 0.25 is counted, and the result is 1. Since 1 is not less than the second sample number threshold of 1, sample 8 is determined to be a redundant sample. Therefore, sample 8 is not added to Set_Core 2, and Set_Core 2 remains unchanged.

[0213] Traverse to sample 9:

[0214] The semantic distance between sample 9 and each sample in Set_Core 2 is calculated. The semantic distance between sample 9 and sample 1 is 0.33, between sample 9 and sample 3 is 0.41, between sample 9 and sample 5 is 0.36, and between sample 9 and sample 7 is 0.29. The number of samples in Set_Core 2 whose semantic distance to sample 8 is less than the redundancy distance threshold of 0.25 is counted, and the result is 0. Since it is less than the second sample number threshold of 1, sample 9 is determined to be a non-redundant sample. Therefore, sample 9 is added to Set_Core 2. At this time, Set_Core 2 = {sample 1, sample 3, sample 5, sample 7, sample 9}.

[0215] Traverse to sample 10:

[0216] Calculate the semantic distance between sample 10 and each sample in the second core training set. The distance between sample 10 and sample 1 is 0.47, the distance between sample 10 and sample 3 is 0.38, the distance between sample 10 and sample 5 is 0.35, the distance between sample 10 and sample 7 is 0.43, and the distance between sample 10 and sample 9 is 0.22. Count the number of samples in Set_Core 2 whose semantic distance to sample 8 is less than the redundancy distance threshold of 0.25, and get 1. Since 1 is not less than the second sample number threshold of 1, sample 10 is determined to be a redundant sample, and therefore sample 10 is not added to Set_Core 2, and Set_Core 2 remains unchanged.

[0217] ④ After the traversal is completed, the final second core training set is obtained.

[0218] The final second core training set is Set_Core2 = {sample 1, sample 3, sample 5, sample 7, sample 9}.

[0219] Starting with the initial 10 samples, the first round of selection was based primarily on semantic coverage gain, aiming to maximize the semantic coverage range. Seven samples were selected to form the first core training set. The second round of selection was based primarily on semantic non-redundancy, aiming to remove samples that were too semantically similar. Five samples were further selected from the seven to form the second core training set. After two rounds of selection, the number of samples was reduced from 10 to 5. The final second core training set has the characteristics of high diversity and low redundancy. Each sample represents an independent semantic category: financial report profitability, market events, asset allocation, credit risk model, and quantitative stock selection model. The second core training set is very suitable for training large financial models, which can improve training efficiency and generalization ability while ensuring the breadth of semantic coverage.

[0220] Additionally, it should be noted that the coverage gain threshold, the first sample number threshold, the redundancy distance threshold, and the second sample number threshold in this embodiment are adjustable parameters. In practical applications, their values ​​can be adjusted according to data characteristics and task requirements.

[0221] The core training set construction method provided in this application has the following advantages:

[0222] Firstly, a balance between screening efficiency and coverage quality is achieved by dynamically controlling the iteration termination through semantic coverage gain. This application introduces semantic coverage gain as the criterion for determining iteration termination. By quantifying the marginal contribution of newly added samples to the semantic coverage of the first core training set, the screening automatically terminates when the semantic coverage of the core training set approaches saturation. When the redundancy of the candidate sample set is high, it can avoid a large number of duplicate samples being included in the core training set, reducing the redundant expansion of training data and lowering computational overhead. When the diversity of the candidate sample set is high, it can ensure that the core training set terminates only after the semantic coverage reaches its optimal level, avoiding the omission of unique semantic samples due to premature termination. Thus, the integrity of semantic coverage is guaranteed while controlling the size of the first core training set.

[0223] Second, a two-stage screening architecture is used to achieve semantic purification and redundancy removal of the core training set. This application adopts a two-stage architecture: iterative screening to construct the first core training set and semantic similarity screening to construct the second core training set. In the first stage, greedy iterative screening quickly covers diverse semantic regions to obtain a first core training set with broad representativeness. In the second stage, based on the semantic similarity between samples, semantically non-redundant samples are selected from the first core training set to obtain a refined second core training set. The two stages work together to form a closed-loop optimization mechanism of "coverage first, redundancy removal later," ensuring that the final core training set has both sufficient semantic coverage and sample conciseness.

[0224] Third, a dual threshold mechanism is used to accurately identify high-value samples and improve the information density of the core training set. In constructing the second core training set, this application introduces a dual judgment mechanism that combines a redundancy distance threshold and a second sample quantity threshold. This mechanism considers not only the semantic similarity between samples but also the clustering density of similar samples. This refined screening strategy can more accurately identify and retain semantically unique high-value samples and eliminate semantically redundant low-value samples, thereby achieving higher information density with the same core training set size.

[0225] Fourth, it provides a high-quality core training set for large models, significantly improving their training performance. The second core training set constructed in this application is used to train large models. This core training set combines conciseness and representativeness. On the one hand, it ensures semantic diversity through dynamic control of semantic coverage gain, ensuring that the model can access a wide range of knowledge dimensions during training. On the other hand, it strictly controls the data scale through redundancy removal, effectively reducing the computational overhead and memory usage of model training, and shortening the training cycle. Ultimately, the second core training set constructed in this application can ensure that large models possess stronger instruction compliance and task generalization capabilities while reducing resource consumption.

[0226] Fifth, it possesses high flexibility and domain adaptability, and can be migrated to various vertical scenarios. The coverage gain threshold, redundancy distance threshold, first sample number threshold, and second sample number threshold in this application can all be flexibly adjusted according to specific business needs and data characteristics. In practical applications, it is only necessary to adjust the coverage gain threshold according to the data distribution characteristics of different vertical domains (such as finance, healthcare, and law) to control the semantic coverage of the first core training set, and adjust the redundancy distance threshold and sample number threshold to control the strictness of redundancy judgment. This allows for rapid reuse in new business scenarios to build high-quality core training sets without the need to redesign complex screening algorithms, significantly reducing the data engineering costs of deploying large-scale models in vertical domains.

[0227] Based on the core training set construction method provided in the above embodiments, this application also provides a large model training method, which may include:

[0228] Step a1: Obtain the target core training set.

[0229] The target core training set is the second core training set constructed using the core training set construction method provided in the above embodiments. Specifically, a first core training set is first constructed from the candidate sample set through iterative screening, with semantic coverage gain as the criterion for termination of the iteration, ensuring that the screening automatically terminates after the semantic coverage of the first core training set reaches its optimal level. Then, from the first core training set, samples that meet the semantic non-redundancy condition are selected based on the semantic similarity between samples through a dual judgment mechanism of redundancy distance threshold and sample number threshold to construct the second core training set. This second core training set is the target core training set. For a more detailed construction process of the core training set, please refer to the above embodiments, which will not be elaborated here.

[0230] Step a2: Train the large model using the target core training set.

[0231] Using the target core training set to train large models (such as supervised training of pre-trained large models) can effectively reduce the storage space, computing power consumption and training time required for training large models, while improving the instruction compliance and task generalization ability of large models.

[0232] This application also provides a core training set construction apparatus, such as... Figure 5 As shown, the core training set construction device may include: a candidate sample set acquisition module 501, a first sample screening module 502, and a second sample screening module 503.

[0233] The candidate sample set acquisition module 501 is used to acquire a candidate sample set, which includes multiple samples.

[0234] The first sample selection module 502 is used to select samples from the candidate sample set and add them to the initially empty first core training set using an iterative selection method until the iteration termination condition is met, thus obtaining the final first core training set. The iteration termination condition is that after the samples are added in the most recent iteration, the semantic coverage gain of the first core training set is less than a preset coverage gain threshold. The semantic coverage gain represents the incremental value brought to the first core training set in terms of semantic coverage by the samples added in the most recent iteration relative to before the samples were added.

[0235] The second sample selection module 503 is used to select semantically non-redundant samples from the final first core training set to construct a second core training set based on the semantic distance between samples. The second core training set is used to train the large model.

[0236] In one possible implementation, when the first sample selection module 502 selects samples from the candidate sample set to add to the initially empty first core training set using an iterative selection method, it is specifically used for:

[0237] Construct an initial empty first core training set;

[0238] One sample is selected from the candidate sample set and added to the first core training set;

[0239] If the current number of samples in the first core training set is less than the preset first sample number threshold, then return to the previous step to select one sample from the candidate sample set and add it to the first core training set.

[0240] If the current number of samples in the first core training set is greater than or equal to the first sample number threshold, then calculate the semantic coverage gain of the first core training set after adding samples this time.

[0241] If the semantic coverage gain is greater than or equal to the preset coverage gain threshold, then return to the previous step and select one sample from the candidate sample set to add to the first core training set.

[0242] If the semantic coverage gain is less than the coverage gain threshold, the iterative selection ends, and the final first core training set is obtained.

[0243] In one possible implementation, when the first sample selection module 502 selects a sample from the candidate sample set to add to the first core training set, it is specifically used for:

[0244] From the samples in the candidate sample set that have not yet been added to the first core training set, select the sample that can maximize the semantic coverage of the first core training set; add the selected sample to the first core training set.

[0245] In one possible implementation, the first sample selection module 502, when calculating the semantic coverage gain of the first core training set after the addition of samples, is specifically used for:

[0246] Calculate the coverage radius of the first core training set after the addition of samples to obtain the first coverage radius, where the first coverage radius characterizes the degree of dispersion of samples in the semantic space of the first core training set after the addition of samples.

[0247] Obtain the coverage radius of the first core training set before adding the sample, and then obtain the second coverage radius;

[0248] Calculate the difference between the first coverage radius and the second coverage radius, and use the calculated difference as the semantic coverage gain of the first core training set after adding samples.

[0249] In one possible implementation, the first sample selection module 502, when calculating the coverage radius of the first core training set after the addition of samples, is specifically used for:

[0250] For each sample in the first core training set after the addition of samples, calculate the semantic distance between the sample and each other sample in the first core training set after the addition of samples, and determine the minimum semantic distance among the calculated semantic distances as the coverage radius of the sample.

[0251] The maximum coverage radius among the coverage radii of all samples in the first core training set after the addition of samples is determined as the coverage radius of the first core training set after the addition of samples.

[0252] In one possible implementation, when the second sample selection module 503 selects semantically non-redundant samples from the final first core training set to construct the second core training set based on the semantic distance between samples, it is specifically used for:

[0253] Construct an initially empty second core training set; select seed samples from the final first core training set according to a preset selection strategy and add them to the second core training set;

[0254] Iterate through all samples in the final first core training set except for the seed sample. For the currently traversed sample, perform the following: calculate the semantic distance between the currently traversed sample and each sample in the second core training set. Based on the calculated semantic distances, determine whether the currently traversed sample is a redundant sample. If the currently traversed sample is determined to be a non-redundant sample, then add the currently traversed sample to the second core training set. After the traversal is completed, the final second core training set is obtained.

[0255] In one possible implementation, the second sample filtering module 503, when determining whether the currently traversed sample is a redundant sample based on the calculated semantic distances, specifically performs the following:

[0256] The system counts the number of samples in the second core training set whose semantic distance from the currently traversed sample is less than a preset redundancy distance threshold. If the counted number of samples is greater than or equal to the preset second sample number threshold, the currently traversed sample is determined to be a redundant sample. If the counted number of samples is less than the second sample number threshold, the currently traversed sample is determined to be a non-redundant sample.

[0257] In one possible implementation, the second sample selection module 503, when selecting seed samples from the final first core training set to add to the second core training set according to a preset selection strategy, is specifically used for:

[0258] Obtain the comprehensive score of each sample in the final first core training set across multiple evaluation dimensions; add the sample with the highest comprehensive score in the final first core training set as a seed sample to the second core training set.

[0259] In one possible implementation, when calculating the semantic distance between the currently traversed sample and each sample in the second core training set, the second sample filtering module 503 is specifically used to calculate the semantic distance between the currently traversed sample and each sample in the second core training set based on the semantic embedding vector of the currently traversed sample and the semantic embedding vector of each sample in the second core training set.

[0260] The core training set construction method apparatus may also include: a semantic embedding vector index list construction module and a semantic embedding vector index addition module.

[0261] The semantic embedding vector index list construction module is used to build a semantic embedding vector index list for the second core training set after adding the seed sample to the second core training set. The initial semantic embedding vector index list contains the semantic embedding vector index corresponding to the seed sample.

[0262] The semantic embedding vector index adding module is used to add the semantic embedding vector index corresponding to the currently traversed sample to the semantic embedding vector index list when it is determined that the currently traversed sample is a non-redundant sample and the currently traversed sample is added to the second core training set.

[0263] The core training set construction device provided in this application can construct a second core training set that is both concise and representative. This second core training set is used for training large models. On the one hand, it can effectively reduce storage space, computing power consumption and training time. On the other hand, it can improve the instruction compliance ability and task generalization ability of large models.

[0264] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor.

[0265] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application; the memory may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage.

[0266] The memory is used to store computer programs, and the processor is used to execute the computer programs so that the electronic device can implement the core training set construction method provided in the above embodiments, or implement the large model training method described above.

[0267] This application also provides a computer storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the core training set construction method provided in the above embodiments, or implement the large model training method described above.

[0268] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the core training set construction method provided in the above embodiments, or to implement the large model training method described above.

[0269] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0270] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0271] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0272] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for constructing a core training set, characterized in that, include: Obtain a candidate sample set, which includes multiple samples; An iterative screening method is used to select samples from the candidate sample set and add them to the initially empty first core training set until the iteration termination condition is met to obtain the final first core training set. The iteration termination condition is that after the samples are added in the most recent iteration, the semantic coverage gain of the first core training set is less than a preset coverage gain threshold. The semantic coverage gain represents the incremental value brought by the samples added in the most recent iteration to the semantic coverage of the first core training set compared to before the samples were added. Based on the semantic distance between samples, semantically non-redundant samples are selected from the final first core training set to construct a second core training set, which is used to train the large model.

2. The core training set construction method according to claim 1, characterized in that, The iterative screening method involves selecting samples from the candidate sample set and adding them to the initially empty first core training set until the iteration termination condition is met, resulting in the final first core training set, including: Construct an initial empty first core training set; Select one sample from the candidate sample set and add it to the first core training set; If the current number of samples in the first core training set is less than the preset first sample number threshold, then return to the previous step to select one sample from the candidate sample set and add it to the first core training set. If the current number of samples in the first core training set is greater than or equal to the first sample number threshold, then calculate the semantic coverage gain of the first core training set after adding samples this time. If the semantic coverage gain is greater than or equal to the preset coverage gain threshold, then return to the process of selecting a sample from the candidate sample set and adding it to the first core training set; If the semantic coverage gain is less than the coverage gain threshold, the iteration ends, and the final first core training set is obtained.

3. The core training set construction method according to claim 2, characterized in that, The step of selecting a sample from the candidate sample set to add to the first core training set includes: From the samples in the candidate sample set that have not yet been added to the first core training set, select the sample that can maximize the semantic coverage of the first core training set. Add the selected samples to the first core training set.

4. The core training set construction method according to claim 2, characterized in that, The calculation of the semantic coverage gain of the first core training set after the addition of samples includes: Calculate the coverage radius of the first core training set after the addition of samples to obtain the first coverage radius, where the first coverage radius characterizes the degree of dispersion of samples in the semantic space of the first core training set after the addition of samples. Obtain the coverage radius of the first core training set before adding the sample, and then obtain the second coverage radius; Calculate the difference between the first coverage radius and the second coverage radius, and use the calculated difference as the semantic coverage gain of the first core training set after adding samples.

5. The core training set construction method according to claim 4, characterized in that, The calculation of the coverage radius of the first core training set after the addition of samples includes: For each sample in the first core training set after the addition of samples, calculate the semantic distance between the sample and each other sample in the first core training set after the addition of samples, and determine the minimum semantic distance among the calculated semantic distances as the coverage radius of the sample. The maximum coverage radius among the coverage radii of all samples in the first core training set after the addition of samples is determined as the coverage radius of the first core training set after the addition of samples.

6. The core training set construction method according to claim 1, characterized in that, The step of selecting semantically non-redundant samples from the final first core training set to construct a second core training set, based on the semantic distance between samples, includes: Construct an initially empty second core training set; According to the preset screening strategy, seed samples are selected from the final first core training set and added to the second core training set; Iterate through all samples in the final first core training set except for the seed samples, and perform the following operations on the currently iterated sample: Calculate the semantic distance between the currently traversed sample and each sample in the second core training set. Based on the calculated semantic distances, determine whether the currently traversed sample is a redundant sample. If the currently traversed sample is determined to be a non-redundant sample, then add the currently traversed sample to the second core training set. After the traversal is completed, the final second core training set is obtained.

7. The core training set construction method according to claim 6, characterized in that, The step of determining whether the currently traversed sample is a redundant sample based on the calculated semantic distances includes: The number of samples in the second core training set whose semantic distance from the currently traversed sample is less than a preset redundancy distance threshold is counted. If the number of samples obtained from the statistics is greater than or equal to the preset second sample number threshold, then the sample currently being traversed is determined to be a redundant sample. If the number of samples obtained from the statistics is less than the second sample number threshold, then the sample currently being traversed is determined to be a non-redundant sample.

8. The core training set construction method according to claim 6, characterized in that, The step of selecting seed samples from the final first core training set and adding them to the second core training set according to a preset screening strategy includes: Determine the comprehensive score of each sample in the final first core training set across multiple evaluation dimensions; The sample with the highest overall score in the final first core training set is used as the seed sample and added to the second core training set.

9. The core training set construction method according to claim 6, characterized in that, The calculation of the semantic distance between the currently traversed sample and each sample in the second core training set includes: Based on the semantic embedding vector of the currently traversed sample and the semantic embedding vector of each sample in the second core training set, calculate the semantic distance between the currently traversed sample and each sample in the second core training set. The core training set construction method also includes: After adding the seed sample to the second core training set, a semantic embedding vector index list for the second core training set is established. The initial semantic embedding vector index list contains the semantic embedding vector index corresponding to the seed sample. If the currently traversed sample is determined to be a non-redundant sample and is added to the second core training set, the semantic embedding vector index corresponding to the currently traversed sample is added to the semantic embedding vector index list so that the semantic embedding vector of each sample in the second core training set can be located through the semantic embedding vector index list when calculating the semantic distance later.

10. A method for training large models, characterized in that, include: Obtain the target core training set, wherein the target core training set is a second core training set constructed using the core training set construction method as described in any one of claims 1 to 9; The large model is trained using the target core training set.

11. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the core training set construction method as described in any one of claims 1 to 9, or to implement the large model training method as described in claim 10.

12. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the core training set construction method as described in any one of claims 1 to 9, or the large model training method as described in claim 10.

13. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the core training set construction method as described in any one of claims 1 to 9, or to implement the large model training method as described in claim 10.

Images