An adaptive normalized mutual information context learning optimization method and system for unbalanced data

Abstract

The application provides an adaptive normalized mutual information context learning optimization method and system for unbalanced data, and relates to the technical field of natural language processing. By constructing a field background pseudo-context to replace an empty string, quantifying inherent bias of the field background, calculating a smoothed normalized mutual information score, applying adaptive regularization weights, and screening optimal example arrangements, the problems of numerical instability, benchmark distortion and unbalanced data in context learning are effectively solved, and the method has the advantages of improving the stability of the model in unbalanced data context learning, avoiding numerical explosion, accurately quantifying the bias of the field background, and adaptively aligning the real label distribution.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to an adaptive normalized mutual information context learning optimization method and system for imbalanced data. Background Technology

[0002] In recent years, large-scale language models have demonstrated significant few-shot learning capabilities in natural language processing tasks. This means that by appending a small number of labeled examples to the test input during the inference phase, the model can generate expected predictions without parameter updates. This contextual learning paradigm effectively reduces the deployment cost for downstream tasks and has become a mainstream research direction. However, its performance exhibits high instability, being extremely sensitive to the selection and order of contextual examples. For the same set of examples, simply adjusting the order can cause the model's prediction accuracy to fluctuate drastically between random guessing and state-of-the-art levels. Key factors contributing to this phenomenon include pre-training bias and the recency effect, where the model tends to repeat labels or high-frequency words appearing at the end of the context. To address this challenge, existing techniques generally introduce calibration mechanisms aimed at quantifying and eliminating the inherent bias of the model in specific contexts.

[0003] Existing research employs a scoring method based on pointwise mutual information, which measures the correlation between the input content and the output label by calculating the ratio of the model's predicted probability to the predicted probability under empty input. Theoretically, this method, by removing baseline bias, can more purely reflect the effective information gain brought by the input. However, in practical applications, it suffers from significant numerical defects: because pointwise mutual information scores are mathematically unbounded, the scores explode dramatically when the baseline bias probability approaches zero. This non-normalization characteristic leads to severe polarization of the model output confidence, causing overconfidence—that is, the model confidence is much higher than the actual prediction accuracy, manifested as a high expected calibration error.

[0004] Existing techniques for obtaining baseline bias terms typically use empty strings or generic placeholders as input to capture the model's prior biases. However, this approach ignores the distribution shift problem. In specialized vertical domains such as finance, healthcare, and computing infrastructure, models exhibit specific semantic sensitivities to domain-specific vocabulary. Empty strings, as out-of-distribution samples, exist in a semantic vacuum and cannot elicit the model's true contextual response level when processing domain-specific text. Distortion in baseline estimation prevents the calculated mutual information from accurately characterizing the model's true preferences in specific domain contexts, thus significantly reducing the effectiveness of contextual example ranking.

[0005] Furthermore, existing context learning ranking optimization methods generally assume a uniform distribution of task labels and tend to select example arrangements that make the prediction results tend towards a uniform distribution. However, in real-world applications such as financial risk control and rare disease detection, data often exhibits severe long-tail distribution characteristics. Forcing the model output to have a uniform distribution violates the true prior distribution of the data, resulting in the model's ability to identify classes with fewer samples being suppressed. Therefore, there is an urgent need for an optimization mechanism that can simultaneously utilize the statistical characteristics of unlabeled data and adaptively align with the true unbalanced prior distribution of the task to solve the problems of distribution bias and numerical stability in example ranking.

[0006] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0007] This application provides an adaptive normalized mutual information context learning optimization method and system for imbalanced data. It has advantages such as improving the stability of context learning, avoiding the numerical explosion problem, accurately quantifying domain background bias, and adaptively aligning with the true imbalanced label distribution.

[0008] Firstly, this application provides an adaptive normalized mutual information context learning optimization method for imbalanced data, which employs the following technical solution:

[0009] An adaptive normalized mutual information context learning optimization method for imbalanced data includes:

[0010] Extract domain vocabulary statistical features from the unlabeled evaluation set, construct a domain background pseudo-context, and use this domain background pseudo-context as an in-distribution probe to replace empty strings;

[0011] The domain background pseudo-context is combined with candidate examples to construct a baseline input, which is then input into a large language model to quantify the inherent bias of the domain background.

[0012] Obtain the prediction probability of the large language model for the real test sample, and calculate the smoothed normalized mutual information score by combining the inherent bias of the domain background.

[0013] An adaptive regularized weight is constructed based on the prior distribution of the true labels of the task, and this weight is applied to the smoothed normalized mutual information score to generate the final calibration score.

[0014] The final calibration score is reconstructed probabilistically, and the optimal example arrangement is selected from the candidate example arrangements through global distribution alignment.

[0015] Optionally, the steps of extracting domain-specific lexical statistical features from the unlabeled evaluation set and constructing a pseudo-context for the domain background include:

[0016] The unlabeled evaluation set is used as a bag-of-words set to establish the empirical word distribution;

[0017] Based on the empirical word distribution, a preset number of domain noise samples are generated through Monte Carlo sampling. All domain noise samples constitute the domain background pseudo-context, and the domain noise samples only retain the statistical features of domain vocabulary and have no syntactic structure and semantic logic of natural language.

[0018] Alternatively, the quantification method for inherent biases in domain context is as follows:

[0019] After filling domain noise samples into a task-specific input template, they are arranged and concatenated with candidate examples to form baseline prompt words;

[0020] The baseline prompt words are input into a large language model to obtain the label prediction probability corresponding to noise samples in each domain.

[0021] The inherent bias of the domain background is obtained by averaging the Monte Carlo integrals of all the predicted probabilities of the labels.

[0022] Optionally, the smoothed normalized mutual information score is calculated as follows:

[0023] By combining the direct posterior probability of the real test sample and the prior marginal probability of the input text from the large language model, and adding a Laplace smoothing term, smooth joint self-information is constructed.

[0024] The difference between the logarithm of the direct posterior probability and the logarithm of the domain background inherent bias is calculated, and the difference is divided by the smoothed joint self-information to obtain the smoothed normalized mutual information score.

[0025] Alternatively, the adaptive regularization weights can be constructed as follows:

[0026] Based on the prior distribution of the true labels in the task, a category scarcity potential is defined that is negatively correlated with the prior probability of the labels.

[0027] The category scarcity potential energy is processed by Boltzmann distribution, normalized using the Softmax function, and the weight distribution is adjusted by introducing a temperature parameter to obtain the adaptive regularized weights.

[0028] The final calibration score is the product of the adaptive regularization weight corresponding to the label and the smooth normalized mutual information score of the label.

[0029] Optionally, the probabilistic reconstruction involves mapping the final calibration score from the feature space to a probabilistic simplex using a Softmax transform with a temperature parameter, resulting in a pseudo-probability distribution.

[0030] The global distribution alignment method is as follows: the global empirical distribution is obtained by arithmetically averaging the pseudo-probability distributions of all samples in the unlabeled evaluation set, and KL divergence is used as the loss function to select the candidate example arrangement that minimizes the KL divergence between the global empirical distribution and the true prior distribution of the task, which is then used as the optimal example arrangement.

[0031] Optionally, when generating domain noise samples, the mean and variance of the length of the real examples are first calculated, and the target length of each domain noise sample is obtained by sampling based on the mean and variance. Then, based on the empirical word distribution, the corresponding target length word sequence is generated by independent and identically distributed sampling to form the domain noise sample.

[0032] Secondly, this application provides an adaptive normalized mutual information context learning optimization system for imbalanced data, comprising:

[0033] The module is used to extract domain vocabulary statistical features from the unlabeled evaluation set, construct a domain background pseudo-context, and use the domain background pseudo-context as an in-distribution probe to replace empty strings;

[0034] The domain context inherent bias module is used to combine the domain context pseudo-context with the candidate example permutation to construct a baseline input, which is then quantified after being input into a large language model to obtain the domain context inherent bias.

[0035] The scoring module is used to obtain the prediction probability of the large language model for the real test sample, and calculate the smoothed normalized mutual information score by combining the inherent bias of the domain background.

[0036] The calibration score module is used to construct an adaptive regularized weight based on the prior distribution of the true labels of the task, and apply the weight to the smoothed normalized mutual information score to generate the final calibration score.

[0037] The output module is used to perform probabilistic reconstruction of the final calibration score and select the optimal example arrangement from the candidate example arrangement through global distribution alignment.

[0038] Thirdly, this application provides a computer device, the device comprising: a memory and a processor, wherein the processor, when executing computer instructions stored in the memory, performs the method described above.

[0039] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described above.

[0040] In summary, this application effectively solves the problems of numerical instability, benchmark distortion, and imbalanced data in context learning by constructing a pseudo-context of the domain background to replace empty strings, quantifying the inherent bias of the domain background, calculating smooth normalized mutual information scores, applying adaptive regularized weights, and selecting the optimal example arrangement. It has the advantages of improving the stability of the model in context learning on imbalanced data, avoiding numerical explosion, accurately quantifying the bias of the domain background, and adaptively aligning with the distribution of the true labels. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiments of this application;

[0042] Figure 2 This is a flowchart illustrating the first embodiment of the adaptive normalized mutual information context learning optimization method for imbalanced data in this application.

[0043] Figure 3 This is a structural block diagram of the first embodiment of the adaptive normalized mutual information context learning optimization system for unbalanced data in this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] Reference Figure 1 , Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiments of this application.

[0046] like Figure 1As shown, the computer device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0047] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0048] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and an adaptive normalized mutual information context learning optimization program for unbalanced data.

[0049] exist Figure 1 In the computer device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in this application can be set in the computer device. The computer device calls the adaptive normalized mutual information context learning optimization program for unbalanced data stored in the memory 1005 through the processor 1001, and executes the adaptive normalized mutual information context learning optimization method for unbalanced data provided in the embodiments of this application.

[0050] Traditional context learning methods suffer from performance fluctuations when handling imbalanced data, exhibiting sensitivity to the selection and order of context examples, and are susceptible to pre-training bias and recency effects. Pointwise mutual information-based scoring methods suffer from numerical unboundedness, confidence polarization, and excessively high confidence levels. Furthermore, existing techniques use empty strings as a baseline bias term, ignoring neighborhood distribution shifts and leading to inaccurate baseline estimation. Simultaneously, existing ranking optimization methods assume uniform label distribution, failing to adapt to the actual imbalanced distribution of data and suppressing the recognition ability of minority classes.

[0051] To address this, embodiments of this application provide an adaptive normalized mutual information context learning optimization method for imbalanced data, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the adaptive normalized mutual information context learning optimization method for unbalanced data in this application.

[0052] In this embodiment, the adaptive normalized mutual information context learning optimization method for imbalanced data includes the following steps:

[0053] Step S10: Extract domain vocabulary statistical features from the unlabeled evaluation set, construct a domain background pseudo-context, and use the domain background pseudo-context as an in-distribution probe to replace the empty string.

[0054] It is understood that, for ease of understanding, some key terms in this embodiment will be explained below:

[0055] An unlabeled evaluation set refers to a dataset that does not contain manually labeled data for a specific task or domain. This set is used to statistically analyze the distribution characteristics of vocabulary within that domain, reflecting its linguistic features.

[0056] Domain-specific lexical statistical features refer to information extracted from unlabeled evaluation sets that reflects statistical attributes such as the frequency of use and co-occurrence relationships of words in a specific domain. These features are used to capture domain-specific language patterns.

[0057] Domain-specific pseudo-context refers to a textual representation that simulates the context of a target domain by aggregating statistical features of domain-specific vocabulary. This pseudo-context aims to provide a baseline environment that conforms to the semantic distribution of the target domain, replacing traditional empty strings or generic placeholders.

[0058] In-distribution probes are inputs used to probe the response patterns of large language models within a specific domain. By using pseudo-contextual information about the domain as an in-distribution probe, the model's actual responses to the domain context can be elicited.

[0059] Candidate example permutation refers to the different combinations and ordering of a small number of "input-output" examples used to guide large language models in prediction during context learning. Different permutations can affect the model's predictive performance.

[0060] The baseline input refers to the text that is input into a large language model after combining the pseudo-context of the domain background with candidate examples to quantify the inherent bias of the model.

[0061] Large language models refer to deep learning models with a large number of parameters, pre-trained on a large amount of text data, capable of understanding and generating natural language and performing various natural language processing tasks.

[0062] Domain-specific bias refers to the prior tendency or predictive preference of a large language model for different labels under pseudo-contexts of a specific domain. This bias reflects the model's default response influenced by the domain context when there is no explicit task input.

[0063] The real test sample refers to the input data that needs to be predicted by a large language model.

[0064] Prediction probability refers to the likelihood that a large language model will output a specific label for a given input sample.

[0065] Smoothed normalized mutual information score is a mutual information metric that has undergone smoothing and normalization. It measures the strength of the association between the real test sample and a specific label, while removing the influence of inherent biases in the domain context and solving the problem of unbounded traditional mutual information values.

[0066] The prior distribution of true labels for a specific task refers to the frequency or proportion of each type of label appearing in the dataset for that particular task. This distribution reflects the imbalanced nature of the data itself.

[0067] Adaptive regularization weights are weighting factors that can be adjusted based on the prior distribution of the true labels for the task. These weights are used to calibrate the smoothed normalized mutual information score to adapt to an imbalanced data distribution.

[0068] The final calibration score is the result of the smoothed normalized mutual information score after applying adaptive regularization weights. This score reflects the association between the sample and the label and takes into account the characteristics of imbalanced data.

[0069] Probabilistic reconstruction refers to the process of mapping the final calibration score from the feature space to the probability space so that it conforms to the characteristics of a probability distribution.

[0070] Global distribution alignment refers to the process of optimizing the arrangement of candidate examples so that the average of the pseudo-probability distributions of all samples is as consistent as possible with the prior distribution of the true labels of the task.

[0071] The optimal example arrangement refers to the combination and sorting of context examples selected through global distribution alignment that enables large language models to exhibit excellent performance on imbalanced data tasks.

[0072] It should be noted that the steps of extracting domain vocabulary statistical features from the unlabeled evaluation set and constructing the domain background pseudo-context include: establishing an empirical word distribution by using the unlabeled evaluation set as a bag-of-words set; generating a preset number of domain noise samples based on the empirical word distribution through Monte Carlo sampling; all domain noise samples constitute the domain background pseudo-context, and the domain noise samples only retain domain vocabulary statistical features and do not have the syntactic structure and semantic logic of natural language.

[0073] Understandably, the aforementioned technical solution ensures that the constructed domain-specific pseudo-context accurately and purely reflects the lexical statistical characteristics of the target domain, avoiding interference from introducing additional syntactic or semantic information that could affect the predictions of large language models. This approach makes the domain-specific pseudo-context a more accurate and representative "in-distribution probe," enabling more precise quantification of the inherent bias of large language models towards specific domain vocabulary. By providing a statistically reliable yet semantically neutral benchmark, this method effectively improves the accuracy and robustness of subsequent smoothed normalized mutual information score calculations, thereby optimizing the performance of the entire adaptive normalized mutual information context learning optimization method. Especially when dealing with imbalanced data, it can more effectively calibrate model predictions and select the optimal example permutation.

[0074] It is understandable that when generating domain noise samples, the mean and variance of the length of the real examples are first calculated, and the target length of each domain noise sample is obtained by sampling based on the mean and variance. Then, based on the empirical word distribution, the corresponding target length word sequence is generated by independent and identically distributed sampling to form the domain noise sample.

[0075] In practical implementation, existing techniques for bias calibration in context learning commonly use empty strings as baseline inputs to estimate the model's inherent bias. However, this approach introduces a significant risk of out-of-distribution bias. Specifically, empty strings constitute a semantic vacuum, lacking both domain-specific lexical features and the ability to simulate the potential activation states induced by domain-specific vocabulary when large language models process specialized texts. To eliminate baseline bias caused by feature mismatch, this step aims to construct a domain-specific pseudo-context. This context must satisfy two orthogonal constraints: first, it must be consistent with real-world domain data in statistical properties; second, it must remain unordered and neutral in semantic logic, thereby achieving an accurate measure of the domain-specific response of large language models.

[0076] First, in order to capture the domain characteristics of the target task, this embodiment starts from the unlabeled evaluation set. Extract full vocabulary statistics. Unlike sequence models that focus on syntactic structure, this step will... It is considered as a bag-of-words set to construct a desemantic global lexical probability space.

[0077] Define the distribution of empirical words Characterized in a specific field In Chinese, any word The prior probability of occurrence. This distribution is obtained by analyzing an unlabeled dataset. The normalized word frequency statistics of all samples were obtained, and its mathematical expression is formalized as follows:

[0078]

[0079] in, This represents a single text sample in the dataset. This indicates the length of the sample. This is an indicator function used for statistical vocabulary. In the sample The formula constructs a static probability field reflecting domain preferences, where any terminology that appears frequently in that domain is considered a probability field. The sampling weights will be high, thus ensuring that the noise generated subsequently has distinct domain attributes.

[0080] Based on the empirical distribution of the construction The process of generating pseudo-demonstrations is defined. To completely remove the semantic logic from the context, an independent and identically distributed sampling strategy is adopted instead of the traditional sentence extraction strategy.

[0081] To simulate the complexity of real input, the average length of real examples is calculated. With variance And sample the target length of the noise sequence from it. Subsequently, a generator containing noise set of samples Each noise sample All by Composed of lexical units from individual samples:

[0082]

[0083] In this formula, Ensure the generated text sequence Composed of disordered word stacking, it disrupts the syntactic structure and logical coherence of natural language; at the same time, its lexical composition strictly follows the statistical laws of the target domain.

[0084] It should be noted that the above technical solution fully considers the length distribution characteristics of real examples when generating domain noise samples. By statistically analyzing the mean and variance of real example lengths, and using this as a basis for sampling, the target length of each domain noise sample is obtained, ensuring that the generated domain noise samples maintain a high degree of consistency with the real data in length. Based on this, independent and identically distributed sampling is performed using empirical word distribution, which generates a sequence of lexical units containing only domain vocabulary statistical features and lacking natural language syntactic structure and semantic logic. This method makes the domain noise samples more representative, and the constructed domain background pseudo-context can more accurately reflect domain features, effectively reducing the bias introduced by mismatched noise sample lengths or unreasonable structures. This further improves the quantification accuracy of inherent biases in the domain background, making the calculation of smoothed normalized mutual information scores more reliable, and ultimately improving the overall performance and robustness of the adaptive normalized mutual information context learning optimization method for imbalanced data.

[0085] Step S20: Combine the pseudo-context of the domain background with the candidate examples to construct a baseline input, and then quantify the inherent bias of the domain background after inputting it into a large language model.

[0086] It should be noted that the quantification method of the inherent bias of the domain background is as follows: after filling the domain noise samples into the task-specific input template, they are arranged and concatenated with the candidate examples to form the baseline prompt words; the baseline prompt words are input into the large language model to obtain the label prediction probability corresponding to each domain noise sample; the Monte Carlo integral is performed on all the label prediction probabilities and the average is obtained to obtain the inherent bias of the domain background.

[0087] In practical implementation, a noise set with domain statistical characteristics is utilized. Arrangement of examples in a specific context Quantitative measurement of inherent biases. From a Bayesian inference perspective, large language models assess labels. Predicted probability It is the input Semantic evidence and arrangement The prior inductions are jointly determined. In order to accurately isolate the components solely determined by permutations... The induced component of the contribution must block the semantic information flow at the input. Since a traditional empty input would cause a distribution shift, this embodiment uses samples without semantic noise. As in-distribution probes, the average response of the model on these probes is used to approximate the permutation. Background activation level in a specific domain environment.

[0088] First, for any given permutation of candidate examples... We construct a set of prompt words for benchmarking. Unlike the standard inference phase, this involves using real queries... As input, this step performs a probe replacement operation, directly calling the probe from samples without semantic noise. ,in Fill it into the task-specific input template. And concatenate it into the context example sequence. after.

[0089] For the Each noise sample, and its corresponding baseline cue word. Defined as:

[0090]

[0091] Here, Indicates sorted Organized k-shot real-world example sequences, This is a sequence concatenation operator. The construction process ensures that the context of the prompt words retains their complete semantic structure, while the input is replaced with words that conform to a domain-experienced word distribution. However, it is a noise stream without logical semantics.

[0092] To build Input parameters are The large language model utilizes the model's autoregressive decoding mechanism to calculate the value of a given current permutation. With noise probe Under the condition of generating candidate tags The conditional probability.

[0093] Considering the time-step dependency of generative models, the labels Consider it as a sequence of lexical units The model's prediction probability under noisy excitation. Strictly defined as the joint likelihood of sequence generation:

[0094]

[0095] in, Represents the first step in the autoregressive generation process. The preceding label prefix. This formula implies that, when faced with meaningless input, large language models are only affected by contextual arrangement. Inertial drive and domain vocabulary The confidence level of the illusion generated by the background color.

[0096] Due to a single noise sample The samples are randomly selected, and the resulting model responses may exhibit local variance. To obtain an unbiased estimate of domain background bias, this step applies the law of large numbers to the set. All conditional probabilities calculated above Perform Monte Carlo integration.

[0097] Arrange For tags Expectation benchmark bias Defined as the marginal likelihood of the model over the neighborhood noise distribution:

[0098]

[0099] The physical meaning of this formula is: Quantification was performed based solely on example arrangement without any valid semantic evidence input. Background noise in the domain, model blindly guesses labels The average probability baseline. This baseline value will be used as the denominator in subsequent steps to calculate the normalized mutual information, and will be used to correct prediction biases caused by model overconfidence.

[0100] It should be noted that, through the above technical solution, this embodiment can overcome the uncertainty problems that may be encountered when quantifying the inherent bias of the domain context. Specifically, by combining domain noise samples with task-specific input templates and candidate example permutations to construct benchmark prompts, the standardization of the input format and the integrity of the context are ensured when evaluating the large language model, enabling the model to exhibit its inherent bias in an environment simulating a real task. Subsequently, by inputting these benchmark prompts into the large language model, the label prediction probabilities of each domain noise sample are obtained, directly capturing the model's original bias in a specific domain context. More importantly, the Monte Carlo integral averaging method is used to process all label prediction probabilities, effectively reducing the impact of the randomness of individual domain noise samples, thereby obtaining a more stable, accurate, and representative inherent bias of the domain context. This quantification method not only improves the robustness of the inherent bias estimation but also provides a reliable benchmark for the subsequent calculation of smoothed normalized mutual information scores, thereby improving the accuracy and stability of the entire adaptive normalized mutual information context learning optimization method, especially when dealing with imbalanced data, it can more effectively calibrate the model's predictions.

[0101] Step S30: Obtain the prediction probability of the large language model for the real test sample, and calculate the smoothed normalized mutual information score by combining the inherent bias of the domain background.

[0102] It should be noted that the smoothed normalized mutual information score is calculated as follows: combining the direct posterior probability of the real test sample and the prior marginal probability of the input text from the large language model, and adding a Laplacian smoothing term to construct smoothed joint self-information; calculating the difference between the logarithm of the direct posterior probability and the logarithm of the inherent bias of the domain background, and dividing the difference by the smoothed joint self-information to obtain the smoothed normalized mutual information score.

[0103] In practice, to further quantify the arrangement of real-world examples relative to baseline bias The provided effective information gain constructs a smooth normalized mutual information measurement mechanism, introduces Laplace smoothing and joint self-information constraints, and strengthens the contextual correlation strength.

[0104] First, the model's response level is measured under conditions containing complete semantic information, using real test samples from an unlabeled dataset. Arrange it with candidate examples Templates are learned and assembled according to standard context. The target label is calculated using the autoregressive generation mechanism of a large model. direct posterior probability :

[0105]

[0106] in, This characterizes the superposition state of effective signal and noise, that is, after receiving dual stimulation from the real context logic guidance and input semantic features, the model reacts to the label. The overall confidence level.

[0107] To map vastly different probability values ​​to a unified metric, a normalization factor is introduced. In information theory, the standard denominator of normalized mutual information is joint self-information. Considering the long-tailed nature of large model prediction distributions, extremely low-probability events may lead to numerical computational instability. This embodiment introduces smoothed joint self-information. And add a regulating factor It is defined as the global self-dissimilarity of the model to the co-occurrence of the "context-input-label" triple:

[0108]

[0109] in, The prior marginal probability of the input text. To prevent An incorrect Laplace smoothing term. This formula sets the maximum information entropy limit for the current semantic combination, which serves as the denominator for subsequent normalization calculations and effectively smooths out numerical noise caused by low-frequency long-tailed samples.

[0110] Finally, integrate benchmark biases. With direct probability Solve for smoothed normalized mutual information scores :

[0111]

[0112] in, It measures the signal-to-noise ratio, eliminating the probability components guessed by the model based solely on domain vocabulary, and accurately extracting the true contextual examples. How much additional semantic gain is provided for prediction? (Denominator term) This suppresses scores generated by high-frequency words while highlighting long-tail correct predictions that, although lower in probability, show a significant improvement over the baseline, thus eliminating the model's overconfidence. It becomes a more robust sorting criterion than the original probability.

[0113] It should be noted that by combining the above technical solution with the direct posterior probability of the real test sample and the prior marginal probability of the input text from a large language model, and introducing a Laplace smoothing term to construct smooth joint self-information, this embodiment effectively solves the zero-frequency problem that may occur in probability estimation and avoids numerical instability caused by logarithmic operations. Based on this, by calculating the difference between the logarithm of the direct posterior probability and the logarithm of the inherent bias of the domain background, and dividing it by the smooth joint self-information, the information association strength between the real test sample and a specific label can be stably and accurately quantified. This calculation method not only improves the robustness of the mutual information score, enabling it to provide reliable evaluations when dealing with imbalanced data or sparse events, but also, through normalization, makes the scores more comparable in different contexts, thus providing a more solid and accurate foundation for subsequent adaptive regularization and optimal example permutation selection.

[0114] Step S40: Construct adaptive regularized weights based on the prior distribution of the true labels of the task, apply the weights to the smoothed normalized mutual information score, and generate the final calibration score.

[0115] It should be noted that the adaptive regularization weights are constructed as follows: a category scarcity potential negatively correlated with the prior probability of the label is defined based on the prior distribution of the true labels of the task; the category scarcity potential is processed by the Boltzmann distribution, normalized using the Softmax function, and a temperature parameter is introduced to adjust the weight distribution to obtain the adaptive regularization weights; the final calibration score is the product of the adaptive regularization weight corresponding to the label and the smooth normalized mutual information score of the label.

[0116] In practice, by using smooth normalized mutual information, the illusionary bias of the model towards domain vocabulary was successfully removed, and confidence polarization at the single-sample level was eliminated, resulting in a set of numerically robust smooth normalized mutual information scores. However, significant statistical drawbacks remain in imbalanced data scenarios. Although smoothed normalized mutual information standardizes the score scale of individual samples, it does not alter the original class density distribution of the dataset. In long-tailed distribution scenarios, the majority class samples, due to their massive number, generate an overwhelming cumulative signal strength during global aggregation statistics. This statistical advantage brought by data scale can easily mask the weak semantic signals of minority classes, causing the model to be insensitive to long-tailed classes in subsequent optimization processes.

[0117] To mitigate the majority class dominance effect caused by skewed data distribution, global prior knowledge of the task is introduced as a regularization term in the context learning scoring mechanism. By constructing a scarcity potential field, higher unit information weights are assigned to long-tail categories, and unbalanced signals are adaptively enhanced within the feature space, ensuring that the semantic contributions of different categories receive fair representation in the overall evaluation.

[0118] First, the information value of each label category from a global task perspective is quantified. Based on the fundamental principles of information theory, low-frequency events often contain higher self-information. This is achieved by utilizing the prior distribution of the task's true labels. ,in This distribution is determined by prior knowledge of the unlabeled dataset, defining the scarcity potential of the categories. :

[0119]

[0120] Prior probability The lower the long-tail category, the greater its corresponding potential energy. The higher the value, the better. This serves as the energy reference for subsequent signal amplification.

[0121] Simple logarithmic potential can lead to an overly drastic or flat weight distribution, failing to adapt to the sensitivities of different models. Therefore, we introduce the Boltzmann distribution form from statistical mechanics to construct adaptive context-learning calibration weights. The potential energy is normalized using the Softmax function, and a temperature parameter is introduced. As a hyperparameter, it is used to adjust the degree of focus on minority class samples:

[0122]

[0123] The entropy value of the control weight distribution, when At that time, the weights tend to focus only on the scarcest category. At this time, the weights tend to be evenly distributed. (Category cardinality) The expected value of the weights is 1, i.e. This ensures that the weighting operation only changes the relative intensity distribution of different types of signals, without compressing or amplifying the absolute numerical scale of the smoothed normalized mutual information score as a whole, thus maintaining the overall numerical stability of the data.

[0124] Finally, the calculated adaptive weights Effect on smoothed normalized mutual information score This generates the final calibration score. :

[0125]

[0126] For a small number of classes, due to The originally weak smoothed normalized mutual information signal is significantly amplified. In the competition for example ranking, even if the model's original prediction confidence for the minority class is not high, as long as it has a positive gain relative to background noise, it can still obtain a higher selection weight after weighting. For the majority class samples, its weights... The signal strength is appropriately attenuated to prevent the model from generating statistical bias due to overfitting high-frequency labels.

[0127] It should be noted that, through the above technical solution, this embodiment can effectively solve the problem that rare classes may be overlooked during the evaluation process in imbalanced data scenarios. The introduction of adaptive regularization weights reasonably amplifies the evaluation scores of rare classes, thus fully reflecting their importance in the final calibration score. This ensures that in the subsequent probabilistic reconstruction and optimal example arrangement selection process, the model can consider all classes more fairly and accurately, avoiding excessive preference for common classes and significantly improving the robustness and effectiveness of context learning optimization on imbalanced data. The introduction of the temperature parameter further provides a flexible adjustment mechanism, which can finely control the influence of rare classes according to specific task requirements, making the calibration process more adaptive and accurate, thereby selecting the optimal example arrangement that better matches the real data distribution.

[0128] Step S50: Probabilistically reconstruct the final calibration score and select the optimal example arrangement from the candidate example arrangements through global distribution alignment.

[0129] Understandably, probabilistic reconstruction involves mapping the final calibration score from the feature space to a probabilistic simplex using a Softmax transformation with a temperature parameter, resulting in a pseudo-probability distribution. The global distribution alignment method is as follows: the global empirical distribution is obtained by arithmetically averaging the pseudo-probability distributions of all samples in the unlabeled evaluation set, and KL divergence is used as the loss function to select the candidate example arrangement that minimizes the KL divergence between the global empirical distribution and the true prior distribution of the task, which is then used as the optimal example arrangement.

[0130] In its implementation, this embodiment establishes baseline bias by constructing a domain noise probe in the preliminary steps, removes confidence illusions using smoothed normalized mutual information, and introduces prior regularization potential to address the long-tail signal overload problem. This generates a calibration score with high robustness and class fairness for any given input sample and candidate permutation. However, micro-calibration at the single-sample level is insufficient to determine the overall quality of the example permutations. The core challenge of context learning lies in finding a universal cue template that possesses optimal generalization ability and distribution adaptability across the entire task domain. Therefore, this step elevates the perspective from micro to macro, performing global distribution alignment to find a permutation structure that maximizes the statistical balance between the induced global predicted distribution and the true prior distribution of the task.

[0131] First, in order to perform statistical evaluation at the distribution level, the calibration score must be... Mapping back to the probabilistic simplex from the feature space. The calibration score is essentially a weighted modulated mutual information intensity and does not possess probabilistic axiomatic properties. Therefore, this embodiment employs a Softmax transform with a temperature parameter to reconstruct the calibration score into a calibrated pseudo-probability distribution. Formally, for any input... and candidate permutation ,Label Reconstruction probability Defined as:

[0132]

[0133] Calibration temperature As a hyperparameter for adjusting the distribution entropy, it controls the sharpness of model decisions. Lower temperatures enhance the confidence of high-scoring categories, while higher temperatures smooth the distribution.

[0134] Subsequently, in order to evaluate the arrangement This embodiment demonstrates performance across the entire task domain using an unlabeled dataset. Calculate the global empirical distribution. This distribution represents the model in a specific permutation. Guided by this approach, the expected aggregation of prediction results for all samples within the domain is performed. Specifically, the labels are obtained by taking the arithmetic mean of the pseudo-probability distributions of all samples in the dataset. Marginal probability estimation:

[0135]

[0136] Finally, this embodiment transforms the example ranking problem into a distribution matching optimization problem. Based on the distribution alignment principle of context learning, a good context structure should induce a predicted distribution that approximates the true prior label distribution of the task as closely as possible. To quantify this approximation, this embodiment uses KL divergence as the loss function to measure the information loss between the global empirical distribution and the true prior distribution. Optimal permutation Defined as the set of all candidate permutations Among them, the permutations that minimize this divergence are:

[0137]

[0138] By solving the above optimization objective, this embodiment forces the model's macroscopic predictive behavior to conform to the prior laws of the objective world. The permutation that minimizes the KL divergence. This means that it not only achieves a high signal-to-noise ratio at the micro level through smoothed normalized mutual information, but also maintains class sensitivity to imbalanced data at the macro level, without experiencing distribution collapse due to the sample size advantage of certain classes. Once selected... This arrangement will be fixed as the final context learning prompt template for subsequent online inference services, thereby ensuring continuous and stable high-performance output.

[0139] It should be noted that, through the above technical solution, this embodiment can transform the discrete final calibration score into a continuous pseudo-probability distribution with probabilistic significance, and flexibly adjust its smoothness through temperature parameters, laying the foundation for subsequent global optimization. Based on this, by arithmetically averaging the pseudo-probability distributions of all samples in the unlabeled evaluation set, a global empirical distribution can be accurately constructed. This distribution comprehensively reflects the model's predictive tendency for the entire evaluation set under the current candidate example arrangement. Furthermore, by using KL divergence as the loss function to align the global empirical distribution with the task's true prior distribution, the difference between the predicted distribution and the true distribution caused by the current example arrangement can be quantified. By minimizing KL divergence, this embodiment can select candidate example arrangements that make the model's predicted distribution closest to the true label prior distribution, thereby effectively avoiding prediction distribution shifts caused by data imbalance or model bias. This ensures that the selected example arrangement can guide the large language model to generate more accurate prediction results that better match the task's true distribution, significantly improving the overall performance and robustness of the adaptive normalized mutual information context learning optimization method for imbalanced data.

[0140] This embodiment effectively solves the problems of numerical instability, benchmark distortion, and imbalanced data in context learning by constructing a pseudo-context of the domain background to replace empty strings, quantifying the inherent bias of the domain background, calculating smooth normalized mutual information scores, applying adaptive regularized weights, and selecting the optimal example arrangement. It has the advantages of improving the stability of the model in context learning on imbalanced data, avoiding numerical explosion, accurately quantifying the bias of the domain background, and adaptively aligning with the distribution of the true labels.

[0141] Furthermore, embodiments of this application also propose a computer-readable storage medium storing a program for adaptive normalized mutual information context learning optimization for imbalanced data. When the program for adaptive normalized mutual information context learning optimization for imbalanced data is executed by a processor, it implements the steps of the method for adaptive normalized mutual information context learning optimization for imbalanced data as described above.

[0142] Reference Figure 3 , Figure 3 This is a structural block diagram of the first embodiment of the adaptive normalized mutual information context learning optimization system for unbalanced data in this application.

[0143] like Figure 3 As shown in the embodiments of this application, the adaptive normalized mutual information context learning optimization system for imbalanced data includes:

[0144] Module 10 is used to extract domain vocabulary statistical features from the unlabeled evaluation set, construct a domain background pseudo-context, and use the domain background pseudo-context as an in-distribution probe to replace empty strings;

[0145] Domain context inherent bias module 20 is used to combine the domain context pseudo-context with the candidate example arrangement to construct a baseline input, which is then quantified after being input into a large language model to obtain the domain context inherent bias.

[0146] The scoring calculation module 30 is used to obtain the prediction probability of the large language model for the real test sample, and calculate the smoothed normalized mutual information score by combining the inherent bias of the domain background.

[0147] The calibration score module 40 is used to construct an adaptive regularized weight based on the prior distribution of the true labels of the task, apply the weight to the smoothed normalized mutual information score, and generate the final calibration score.

[0148] Output module 50 is used to perform probabilistic reconstruction of the final calibration score and select the optimal example arrangement from the candidate example arrangement through global distribution alignment.

[0149] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solution of this application. In specific applications, those skilled in the art can make settings as needed, and this application does not impose any restrictions on this.

[0150] This embodiment effectively solves the problems of numerical instability, benchmark distortion, and imbalanced data in context learning by constructing a pseudo-context of the domain background to replace empty strings, quantifying the inherent bias of the domain background, calculating smooth normalized mutual information scores, applying adaptive regularized weights, and selecting the optimal example arrangement. It has the advantages of improving the stability of the model in context learning on imbalanced data, avoiding numerical explosion, accurately quantifying the bias of the domain background, and adaptively aligning with the distribution of the true labels.

[0151] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0152] In addition, for technical details not described in detail in this embodiment, please refer to the method for adaptive normalized mutual information context learning optimization for unbalanced data provided in any embodiment of this application, which will not be repeated here.

[0153] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0154] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application. The above are only preferred embodiments of this application and do not limit the patent scope of this application. All equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An adaptive normalized mutual information context learning optimization method for imbalanced data, characterized in that, include: Extract domain vocabulary statistical features from the unlabeled evaluation set, construct a domain background pseudo-context, and use this domain background pseudo-context as an in-distribution probe to replace empty strings; The domain background pseudo-context is combined with candidate examples to construct a baseline input, which is then input into a large language model to quantify the inherent bias of the domain background. Obtain the prediction probability of the large language model for the real test sample, and calculate the smoothed normalized mutual information score by combining the inherent bias of the domain background. An adaptive regularized weight is constructed based on the prior distribution of the true labels of the task, and this weight is applied to the smoothed normalized mutual information score to generate the final calibration score. The final calibration score is reconstructed probabilistically, and the optimal example arrangement is selected from the candidate example arrangement through global distribution alignment. The smoothed normalized mutual information score is calculated as follows: By combining the direct posterior probability of the real test sample and the prior marginal probability of the input text from the large language model, and adding a Laplace smoothing term, smooth joint self-information is constructed. Calculate the difference between the logarithm of the direct posterior probability and the logarithm of the domain background inherent bias, and divide the difference by the smoothed joint self-information to obtain the smoothed normalized mutual information score. The adaptive regularization weights are constructed as follows: Based on the prior distribution of the true labels in the task, a category scarcity potential is defined that is negatively correlated with the prior probability of the labels. The category scarcity potential energy is processed by Boltzmann distribution, normalized using the Softmax function, and the weight distribution is adjusted by introducing a temperature parameter to obtain the adaptive regularized weights. The final calibration score is the product of the adaptive regularization weight corresponding to the label and the smooth normalized mutual information score of the label. Among them, probabilistic reconstruction is to map the final calibration score from the feature space to the probabilistic simplex through a Softmax transformation with temperature parameters, thereby obtaining a pseudo-probability distribution; The global distribution alignment method is as follows: the global empirical distribution is obtained by arithmetically averaging the pseudo-probability distributions of all samples in the unlabeled evaluation set, and KL divergence is used as the loss function to select the candidate example arrangement that minimizes the KL divergence between the global empirical distribution and the true prior distribution of the task, which is then used as the optimal example arrangement.

2. The method as described in claim 1, characterized in that, The steps for extracting domain-specific lexical statistical features from an unlabeled evaluation set and constructing a pseudo-context for the domain background include: The unlabeled evaluation set is used as a bag-of-words set to establish the empirical word distribution; Based on the empirical word distribution, a preset number of domain noise samples are generated through Monte Carlo sampling. All domain noise samples constitute the domain background pseudo-context, and the domain noise samples only retain the statistical features of domain vocabulary and have no syntactic structure and semantic logic of natural language.

3. The method as described in claim 1, characterized in that, The method for quantifying inherent biases based on domain background is as follows: After filling domain noise samples into a task-specific input template, they are arranged and concatenated with candidate examples to form baseline prompt words; The baseline prompt words are input into a large language model to obtain the label prediction probability corresponding to noise samples in each domain. The inherent bias of the domain background is obtained by averaging the Monte Carlo integrals of all the predicted probabilities of the labels.

4. The method as described in claim 1, characterized in that, When generating domain noise samples, the mean and variance of the length of real examples are first calculated. Based on the mean and variance, the target length of each domain noise sample is obtained by sampling. Then, based on the empirical word distribution, the corresponding target length word sequence is generated by independent and identically distributed sampling to form the domain noise sample.

5. An adaptive normalized mutual information context learning optimization system for imbalanced data, characterized in that, Performing the method as described in claim 1 includes: The module is used to extract domain vocabulary statistical features from the unlabeled evaluation set, construct a domain background pseudo-context, and use the domain background pseudo-context as an in-distribution probe to replace empty strings; The domain context inherent bias module is used to combine the domain context pseudo-context with the candidate example permutation to construct a baseline input, which is then quantified after being input into a large language model to obtain the domain context inherent bias. The scoring module is used to obtain the prediction probability of the large language model for the real test sample, and calculate the smoothed normalized mutual information score by combining the inherent bias of the domain background. The calibration score module is used to construct an adaptive regularized weight based on the prior distribution of the true labels of the task, and apply the weight to the smoothed normalized mutual information score to generate the final calibration score. The output module is used to perform probabilistic reconstruction of the final calibration score and select the optimal example arrangement from the candidate example arrangement through global distribution alignment.

6. A computer device, characterized in that, The device includes a memory and a processor, wherein the processor, when executing computer instructions stored in the memory, performs the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 4.

Images