Model training loss determination method and model training method

Abstract

This application discloses a method for determining model training loss and a model training method, relating to the field of computing technology. The method includes: obtaining a current sample from training sample sets that are respectively matched to various server operating states; determining the sample similarity between the current sample and at least one reference sample in the test sample set, and obtaining the marginal distribution similarity between the current sample and the training sample set and the test sample set; determining a similarity weight matching the current sample based on at least one sample similarity and the marginal distribution similarity; determining a target sample loss value based on the similarity weight and the loss value corresponding to the current sample, wherein the target sample loss value is used to update the model parameters in the current model, and then dynamically adjusting the sample loss through sample difference. This solves the technical problem of poor model training effect in related technologies and improves the model training effect.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for determining model training loss and a model training method. Background Technology

[0002] Reinforcement learning, an important branch of machine learning, aims to learn optimal behavioral policies by maximizing cumulative rewards through the interaction between an agent and its environment. With the widespread application of multimodal large models in complex tasks, reinforcement learning has become a core technology for improving the quality of model generation and task adaptability.

[0003] However, reinforcement learning training of large multimodal models generally faces a serious domain adaptation problem: the training and test sets differ significantly in data distribution, specifically in edge distribution and textual differences. This distribution difference causes the model to perform well on the training set, but its generalization performance drops sharply in real-world testing scenarios. In other words, existing model training methods suffer from poor training results when faced with significant differences in data distribution.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method for determining model training loss and a model training method to at least address the problem of poor model training performance in related technologies.

[0006] According to one aspect of the embodiments of this application, a method for determining model training loss is provided, comprising: obtaining a current sample from a training sample set that is respectively matched with multiple server operating states; determining the sample similarity between the current sample and at least one reference sample in a test sample set, and obtaining the marginal distribution similarity between the current sample and the training sample set and the test sample set, wherein the training samples in the training sample set and the test samples in the test sample set include multiple operating index parameter values ​​that match the server operating states, and the marginal distribution similarity is determined based on the probability distribution difference between the training sample set and the test sample set in at least one feature dimension; determining a similarity weight matching the current sample based on at least one sample similarity and the marginal distribution similarity; and determining a target sample loss value based on the similarity weight and the loss value corresponding to the current sample, wherein the target sample loss value is used to update the model parameters in the current model.

[0007] According to another aspect of the embodiments of this application, a model training method is provided, comprising determining target sample loss values ​​corresponding to multiple training samples in a training sample set that are respectively matched with multiple server operating states, wherein the training samples include multiple operating index parameter values ​​that match the server operating states, the target sample loss value is determined based on the similarity weight matching the training sample and the loss value corresponding to the training sample, and the similarity weight is determined based on the sample similarity between the training sample and at least one reference sample in the test sample set; determining a training loss value matching the current training stage based on the summation result of the target sample loss values ​​corresponding to the multiple training samples and the policy entropy matching the current model; and determining the current model as the target model for completing training if the training loss value meets the target condition.

[0008] According to another aspect of the embodiments of this application, an apparatus for determining model training loss is also provided, comprising: an acquisition unit for acquiring a current sample from training sample sets that are respectively matched with multiple server operating states; a first determination unit for determining the sample similarity between the current sample and at least one reference sample in a test sample set, and acquiring the marginal distribution similarity between the current sample and the training sample set and the test sample set, wherein the training samples in the training sample set and the test samples in the test sample set include multiple operating index parameter values ​​that match the server operating states, and the marginal distribution similarity is determined based on the probability distribution difference between the training sample set and the test sample set in at least one feature dimension; a second determination unit for determining a similarity weight matching the current sample based on at least one sample similarity and the marginal distribution similarity; and a third determination unit for determining a target sample loss value based on the similarity weight and the loss value corresponding to the current sample, wherein the target sample loss value is used to update the model parameters in the current model.

[0009] According to another aspect of the embodiments of this application, a device for determining model training loss is also provided, comprising: a fourth determining unit, for determining target sample loss values ​​corresponding to multiple training samples in a training sample set that are respectively matched with multiple server operating states, wherein the training samples include multiple operating index parameter values ​​that match the server operating states, and the target sample loss value is determined based on the similarity weight matched with the training sample and the loss value corresponding to the training sample, wherein the similarity weight is determined based on the sample similarity between the training sample and at least one reference sample in the test sample set; a fifth determining unit, for determining a training loss value matched with the current training stage based on the summation result of the target sample loss values ​​corresponding to the multiple training samples and the policy entropy matched with the current model; and a sixth determining unit, for determining the current model as the target model for completing training if the training loss value meets the target condition.

[0010] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of determining the training loss of any of the above-described models.

[0011] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method for determining the training loss of any of the above-described models.

[0012] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of determining the training loss of any of the above-described models.

[0013] Through the above-described embodiments of this application, by combining sample similarity and marginal distribution similarity to determine similarity weights, the loss values ​​of training samples are adaptively weighted to form the target sample loss value to update model parameters, effectively alleviating the problem of decreased generalization performance caused by the distribution difference between the training set and the test set.

[0014] Specifically, sample similarity identifies the individual correlation between the current training samples and test samples at the micro level, while marginal distribution similarity quantifies the overall probability distribution difference between the training set and the test set in terms of feature dimensions at the macro level. The similarity weight determined by the fusion of the two can accurately reflect the importance of each training sample relative to the test scenario. For samples that differ significantly from the test distribution, their weights are reduced, while for samples that are consistent with the test distribution, their weights are increased. This makes the model parameter update process more focused on the effective data distribution in the test scenario, thereby narrowing the marginal distribution gap between training and testing, improving the model's generalization performance in real test scenarios, and solving the technical problem of poor training effect in existing technologies. Attached Figure Description

[0015] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a hardware structure block diagram of a method for determining model training loss according to an embodiment of this application;

[0017] Figure 2 This is a flowchart of a method for determining model training loss according to an embodiment of this application;

[0018] Figure 3 This is a flowchart of a model training method according to an embodiment of this application;

[0019] Figure 4 This is a flowchart of another method for determining model training loss according to an embodiment of this application;

[0020] Figure 5 This is a structural block diagram of a device for determining model training loss according to an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0022] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] The specific application environment architecture or specific hardware architecture on which the method for determining the model training loss depends is described here.

[0025] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of an embodiment of the present application for implementing a method for determining model training loss. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0026] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the method for determining model training loss in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0027] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0028] The embodiments of this application provide a method for determining model training loss. The method is described in detail below in conjunction with the execution flow of the above-described method for determining model training loss.

[0029] This embodiment provides a method for determining model training loss. Figure 2 This is a flowchart of a method for determining model training loss according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0030] S202, Obtain the current sample from the training sample set that matches each of the various server operating states;

[0031] S204, determine the sample similarity between the current sample and at least one reference sample in the test sample set, and obtain the marginal distribution similarity between the current sample and the test sample set. The training samples in the training sample set and the test samples in the test sample set include multiple operating index parameter values ​​that match the server's operating status. The marginal distribution similarity is determined based on the difference in probability distribution between the training sample set and the test sample set in at least one feature dimension.

[0032] S206, determine the similarity weight matching the current sample based on at least one sample similarity and marginal distribution similarity;

[0033] S208. Determine the target sample loss value based on the similarity weight and the loss value corresponding to the current sample. The target sample loss value is used to update the model parameters in the current model.

[0034] In the above embodiments of this application, the current model in the training state can be a multimodal model, used to perform data processing tasks in one or more technical scenarios, including but not limited to server intelligent operation and maintenance scenarios, computer vision scenarios, natural language processing scenarios, and reinforcement learning scenarios.

[0035] For example, in a server intelligent operation and maintenance scenario, the current model can be a multimodal server health status classification network (such as a fusion architecture of graph neural networks and temporal convolution, GNN-TCN). Correspondingly, the training sample set can include real-world production environment observation data (such as time-series metrics, logs, recorded physical temperature and power consumption, and real fault events from actual data centers), while the test sample set can include simulated synthetic operation and maintenance data (such as simulated server load fluctuations). Furthermore, after training the current model based on the training sample set and testing it using the test sample set, the current model can be used to intelligently classify the server's operating status (such as "normal," "resource overload," "software leak," "hardware heat dissipation failure") or automatically identify the root cause of anomalies, thereby achieving automated operation and maintenance decisions without manual rule intervention.

[0036] For example, in a computer vision scenario, the current model could be an image classification network (such as ResNet50) or an object detection network (such as YOLO). Correspondingly, the training sample set could include rendered images from the synthetic domain (such as simulated driving scenarios), and the test sample set could include images captured by a camera from the real domain. Furthermore, after training the current model based on the training sample set and testing it against the test sample set, the current model can be used to detect or classify objects within an image.

[0037] For example, in a natural language processing scenario, the current model can be a large language model (such as LLaMA) or a text understanding network (such as BERT). Correspondingly, the training sample set can include general-domain text corpora (such as encyclopedic texts or general medical descriptions), while the test sample set can include domain-specific professional texts (such as medical records). Furthermore, after training the current model based on the training sample set and testing it on the test sample set, the current model can be used for semantic understanding of medical texts or extraction of medical record information.

[0038] For example, in reinforcement learning scenarios, the current model can be a policy network (such as the Actor network in the PPO algorithm) or a decision network. Correspondingly, the training sample set can include interaction trajectory data from a simulation environment (such as state-action-reward sequences or empirical data under simulation conditions), while the test sample set can include operation trajectories from a real physical environment. Furthermore, after training the current model based on the training sample set and testing it against the test sample set, the current model can be used to make behavioral decisions or control the motion of an agent.

[0039] The above application scenarios and model types are merely illustrative examples and do not limit the specific type of the current model in the embodiments of this application.

[0040] It should be further noted that in the above steps S202 and S204, the server operating status includes, for example, a downtime state, an overload state, and a healthy state; the current sample can be a single training instance obtained by batch sampling from the training sample set; the reference sample can be a representative sample selected from the test sample set to establish a mapping relationship between the training data and the test data, for example, it can be determined by cluster centers or random sampling.

[0041] Furthermore, the sample similarity in step S204 above can be used to measure the proximity of a single training sample and a test sample in the feature space. For example, it can be represented by cosine similarity or Euclidean distance to reflect the domain correlation at the individual level. The above-mentioned multiple operational indicator parameters include time-series values ​​(a sequence of values ​​that change over time, such as CPU utilization, memory usage, disk IOPS, network throughput, and number of processes), semantics based on log analysis, physical status indicators (server rack temperature, intake / exhaust air temperature, humidity, fan speed, power consumption, etc.), and information obtained from the analysis of collected images / videos (server indicator light status, hard drive activity light flashing frequency, and physical displacement / obstruction of equipment collected by data center inspection cameras).

[0042] The aforementioned marginal distribution similarity can be used to quantify the overall probability distribution difference between the training set and the test set across specific feature dimensions. For example, when both the current sample and the reference sample include image objects, this overall probability distribution difference can be represented by the color histogram of the image; when both the current sample and the reference sample include text objects, this overall probability distribution difference can be represented by the word frequency distribution of the text. Optionally, the aforementioned marginal distribution similarity can be calculated using methods such as CORAL (covariance alignment) or MMD (maximum mean difference) to reflect the degree of domain offset at a macroscopic level.

[0043] It should be noted that the similarity weights obtained in step S206 above can be adaptive coefficients used to fuse individual similarity and distribution similarity, thereby adjusting the contribution ratio of the current sample in gradient update.

[0044] It should be noted that, in an optional implementation, in step S208 above, the similarity weight can be directly multiplied by the loss value of the current sample to obtain the loss value of the target sample.

[0045] In another alternative implementation, the similarity weights can be further processed to obtain other forms of weight coefficients, and then directly multiplied with the loss value of the current sample to obtain the loss value of the target sample.

[0046] In another alternative implementation, the similarity weights can be further processed to obtain other forms of weight coefficients, and then combined with the similarity weights to obtain the target sample weights. Finally, the target sample loss value is obtained by directly multiplying the target sample weights by the loss value of the current sample.

[0047] The loss value for the current sample can be, but is not limited to, cross-entropy loss or Clipped Surrogate Loss (PPO) of Proximal Policy Optimization, which is then used to update the model parameters through backpropagation. For example, if the current model is ResNet, the convolutional kernels of ResNet can be updated; if the current model is a Transformer model, the attention weights of the Transformer model can be updated.

[0048] Through the above-described embodiments of this application, the current model can adaptively identify the distribution relationship between training data and target test data when updating parameters, assign high weights to samples within the distribution to make full use of effective information, and assign low weights to samples outside the distribution to suppress domain offset interference, thereby improving the cross-domain generalization ability of the model when there are marginal distribution differences between the training set and the test set.

[0049] In an optional implementation, step S208 above, determining the target sample loss value based on the similarity weight and the loss value corresponding to the current sample, includes:

[0050] S208-1, determine the stage adaptation weights based on similarity weights and the current training stage of the model;

[0051] S208-2, determine at least one reference weight, and determine the target sample weight that matches the current sample based on the at least one reference weight and the stage adaptation weight;

[0052] S208-3, determine the target sample loss value based on the product of the target sample weight and the loss value corresponding to the current sample.

[0053] It is understood that, in the above embodiments of this application, the similarity weight can be further combined with the training phase to determine the phase adaptation weight, and combined with at least one reference weight to determine the target sample weight that matches the current sample.

[0054] The following section provides a further explanation of the method for determining the adaptation weights for the aforementioned stages.

[0055] In an optional implementation, S208-1 above, determining the stage adaptation weight based on the similarity weight and the current training stage of the model, includes:

[0056] S208-11, the ratio between the current number of training steps and the total number of training steps is determined as the stage description coefficient;

[0057] It should be noted that the current training steps mentioned above can be the cumulative number of parameter updates t performed from the start of training to the current moment, and the total training steps can be the total number of parameter updates T to be performed within the preset complete training cycle. Therefore, the stage description coefficient can be described as... , Used to indicate training progress.

[0058] S208-12, Process the stage description coefficients according to the first target activation function to obtain the first stage feature coefficients;

[0059] S208-13, the product between the first-stage feature coefficients and the similarity weights is determined as the first adaptation weight;

[0060] S208-14, The stage description coefficients are processed according to the second objective activation function to obtain the second stage feature coefficients;

[0061] S208-15, the product between the second-stage feature coefficients and the reference similarity weights is determined as the second adaptation weights, wherein the sum of the reference similarity weights and the similarity weights is 1, and the sum of the first target activation function and the second target activation function is 1;

[0062] S208-16, the sum of the first-stage characteristic coefficients and the second-stage characteristic coefficients is determined as the stage adaptation weight.

[0063] It should be noted that, in the above implementation, the reference similarity weight and the similarity weight can be two complementary values; and the first target activation function and the second target activation function can be two activation functions with a complementary relationship.

[0064] The following is a detailed explanation of the first objective activation function mentioned above:

[0065] The above process of the stage description coefficients based on the first objective activation function yields the first stage feature coefficients, including:

[0066] The difference between the stage description coefficient and the offset parameter is used as the first intermediate coefficient;

[0067] The product of the first intermediate coefficient and the smoothing parameter is determined as the second intermediate coefficient;

[0068] The activation result of the second intermediate coefficient is determined as the first-stage characteristic coefficient.

[0069] Specifically, in the above description of offset parameters The smoothing coefficient is described as In this case, the first objective activation function can be expressed as:

[0070]

[0071] At the same time, it can be simplified to describe as:

[0072]

[0073] in, That is, the Sigmoid activation function. , This is the transition threshold (the default value can be 0.5).

[0074] Correspondingly, the second objective activation function mentioned above can be expressed as:

[0075]

[0076] At the same time, it can be simplified to describe as:

[0077]

[0078] Having determined the first and second target activation functions, and with the similarity weights represented as follows: In this case, the stage adaptation weights corresponding to the current sample It can be determined in the following way:

[0079]

[0080] in, In the early stages of training hour, High proportion, with greater weight given to differential data; later stages of training hour, The higher the proportion, the greater the weight of similar data.

[0081] In the above implementation, stage-adaptive weights are adopted to non-linearly couple normalized training progress with sample comprehensive similarity, and the Sigmoid function is used to achieve a smooth transition of training strategy: in the early stage of training, in response to a low progress ratio, the weights of samples that differ significantly from the test distribution are automatically increased to enhance the model's robustness in learning out-of-distribution features; in the later stage of training, in response to a high progress ratio, the weights of samples similar to the test distribution are automatically increased to refine the decision boundary and accelerate convergence; thus, the training oscillation problem caused by hard stage switching in the traditional model training process is effectively avoided, and the generalization performance and convergence accuracy of the model in cross-domain scenarios are significantly improved while maintaining the stability of strategy optimization.

[0082] The following further explains the method for determining at least one reference weight in S208-2 above. In an optional embodiment, determining at least one reference weight includes at least one of the following:

[0083] Method 1: Use similarity weight as the first reference weight;

[0084] In Method 1, the similarity weight can be further used as a reference weight to determine the target sample weight. In Method 1, the similarity weight determined based on marginal distribution similarity and sample similarity can be directly used as the first reference weight. This weight characterizes the degree of matching between the current training sample and the target test distribution from a domain adaptation perspective. By introducing this weight into the target sample loss calculation, the model parameter update process prioritizes samples consistent with the test domain distribution, thereby alleviating the domain shift problem between the training set and the test set and improving the model's cross-domain generalization ability.

[0085] Method 2: Obtain the difficulty coefficient matching the current sample; determine the second reference weight based on the difficulty coefficient, where the difficulty coefficient is used to indicate the reasoning difficulty corresponding to the current sample;

[0086] In Method 2, the inference difficulty coefficient of the current sample can be obtained (e.g., determined based on the model prediction loss value, prediction confidence, or gradient magnitude), and a second reference weight can be determined based on this difficulty coefficient. This weight represents the degree of difficulty of the current sample for model learning from the perspective of sample attributes. By responding to a high difficulty coefficient by increasing the weight of the corresponding sample or dynamically adjusting the difficulty sensitivity based on the temperature annealing mechanism, the model can adaptively achieve course learning from easy to difficult or focusing on difficult samples during the training process, thereby enhancing the model's ability to learn complex patterns.

[0087] Method 3: Obtain the policy entropy that matches the current model, and determine the third reference weight based on the policy entropy. The policy entropy is used to indicate the degree of dispersion of the current model's output distribution.

[0088] In Method 3, the policy entropy of the current policy network (i.e., the Shannon entropy of the policy output action probability distribution) can be obtained, and a third reference weight can be determined based on the policy entropy. This weight represents the exploration-exploitation balance of the current policy from the perspective of model state. By responding to high policy entropy (discrete distribution, overexploitation) to increase the weight of samples similar to the test distribution to stabilize the gradient, or responding to low policy entropy (concentrated distribution, overexploitation) to increase the weight of dissimilar samples to restore exploration, the adaptive adjustment of policy uncertainty during training can be achieved, preventing the policy from converging to a local optimum too early.

[0089] It should be noted that, in addition to using the three methods mentioned above to determine the reference weights separately, multiple methods can be combined and the corresponding target sample weights can be determined by weighted summation.

[0090] In one optional implementation, all three methods described above can be used simultaneously to obtain the corresponding target sample weights. In this implementation, the second reference weight is characterized as... The third reference weight is represented as The first reference weight is represented as In the case of target sample weights It can be determined in the following way:

[0091]

[0092] in, , .

[0093] In another optional implementation, methods two and three described above can be used simultaneously to obtain the corresponding target sample weights. In this implementation, the target sample weights... It can be determined in the following way:

[0094]

[0095] in, , .

[0096] The method for determining the second reference weight is further explained below. In one optional implementation, determining the second reference weight based on the difficulty coefficient includes:

[0097] The temperature coefficient is determined based on the training phase, and the temperature coefficient is positively correlated with the training phase.

[0098] The product of the temperature coefficient and the difficulty coefficient is determined as the third intermediate coefficient;

[0099] The second reference weight, which matches the third intermediate coefficient, is determined based on the exponential response function.

[0100] The method for determining the aforementioned temperature coefficient may include the following steps:

[0101] The ratio between the current number of training steps and the total number of training steps is determined as the stage description coefficient;

[0102] The product between the stage description coefficient and the temperature difference is determined, and the sum of the product and the initial temperature coefficient is determined as the temperature coefficient, wherein the temperature difference is determined based on the difference between the maximum temperature coefficient and the initial temperature coefficient.

[0103] Specifically, the stage description coefficient can be described as the ratio between the current training step t and the total training steps T. Temperature coefficient It can be determined in the following way:

[0104]

[0105] in, This is the initial temperature (which can be set to 0.5 by default). This represents the maximum temperature (which can be set to 2.0 by default). Therefore, the above temperature coefficient... It can increase linearly with the training progress.

[0106] Furthermore, after determining the aforementioned temperature coefficient Subsequently, the aforementioned second reference weight It can be determined in the following way:

[0107]

[0108] in, This represents the difficulty level of the current sample.

[0109] Through the above-described implementation method of this application, a linear annealing mechanism based on the temperature coefficient is used to achieve dynamic perception and adaptive adjustment of sample difficulty. In the early stages of training, the lower temperature coefficient keeps the weight function in an approximately linear region, resulting in small differences in weights between samples. The model focuses on learning low-difficulty data to establish basic decision-making capabilities. As the training progresses, the temperature coefficient gradually increases to its maximum value, the weight function enters the saturation region, high-difficulty data receives significantly increased weights, and low-difficulty data receives correspondingly reduced weights. This guides the model to gradually tackle complex samples after mastering basic patterns, effectively matching the evolution of the model's learning ability and improving final convergence performance and generalization ability.

[0110] The following is a list of difficulty levels The method for determining this will be explained. In one optional implementation, before obtaining the difficulty coefficient for matching the current sample, the method further includes:

[0111] The current sample is repeatedly input into the reference inference model;

[0112] Obtain the reasoning success rate of completing correct reasoning operations using the reference reasoning model;

[0113] The difficulty coefficient is determined based on the reasoning success rate, and there is a negative correlation between the reasoning success rate and the difficulty coefficient.

[0114] In the above implementation, in order to obtain the difficulty coefficient matching the current sample, the current sample is first repeatedly input into the reference inference model, and the inference success rate of the model to complete the correct inference operation for the current sample is obtained through multiple forward propagations. Then, the difficulty coefficient is determined based on the inference success rate, wherein there is a negative correlation between the inference success rate and the difficulty coefficient, that is, the lower the inference success rate, the more difficult it is for the reference model to process the current sample correctly, and the higher the corresponding difficulty coefficient.

[0115] Furthermore, in conjunction with the specific implementation method of data difficulty assessment, the above-mentioned reference inference model is preferably a small pre-trained model with the same architecture as the target model. After fine-tuning, this model has basic inference capabilities, but the number of parameters and computational complexity are significantly lower than the target model, so as to objectively reflect the inherent difficulty attributes of the sample while ensuring assessment efficiency.

[0116] In the specific evaluation process, for each sample in the training set Input it into the aforementioned small pre-trained model and perform K inferences (e.g., K=8, count the number of correct answers). The difficulty coefficient is calculated based on the ratio of the number of correct answers to the total number of reasoning attempts. .

[0117] Through the above implementation methods, objective quantification and automatic evaluation of the difficulty of training samples are achieved. This implementation method utilizes the inference uncertainty of the reference model as a measure of difficulty, which can more accurately reflect the true difficulty of the samples relative to the current model's learning ability. In particular, by statistically analyzing the success rate through multiple inferences, the interference of model randomness (such as randomness caused by Dropout or batch normalization) on the difficulty evaluation is effectively reduced, improving the stability and reliability of the difficulty coefficient. Furthermore, using a small pre-trained model with the same architecture as the target model as a reference ensures the matching of the difficulty evaluation with the ability of the subsequent training process. This allows the difficulty adaptive weights determined based on the difficulty coefficient to more accurately guide the model to focus on easy-to-learn samples in the early stages of training and difficult samples in the later stages, thereby optimizing the model training effect and improving the model convergence efficiency and final performance.

[0118] The method for determining the third reference weight is further explained below. In an optional implementation, obtaining the policy entropy that matches the current model and determining the third reference weight based on the policy entropy includes:

[0119] The current training step number is determined as the reference stage descriptor coefficient;

[0120] The reference stage descriptor coefficients are processed according to the third objective activation function to obtain the first entropy feature coefficients;

[0121] The product of the first entropy feature coefficient and the similarity weight is determined as the first entropy reference weight;

[0122] The reference stage descriptor coefficients are processed according to the fourth objective activation function to obtain the second entropy feature coefficients;

[0123] The product of the second entropy feature coefficient and the reference similarity weight is determined as the second entropy reference weight, wherein the sum of the reference similarity weight and the similarity weight is 1, and the sum of the third target activation function and the third target activation function is 1;

[0124] The sum of the first entropy characteristic coefficient and the second entropy characteristic coefficient is determined as the third reference weight.

[0125] It should be noted that, in the above implementation, the reference similarity weight and the similarity weight can be two complementary values; and the third target activation function and the fourth target activation function can be two activation functions with a complementary relationship.

[0126] The third objective activation function is explained in detail below. The reference stage descriptive coefficients are processed using the third objective activation function to obtain the first entropy feature coefficients, including:

[0127] The difference between the current policy entropy determined based on the stage description coefficient and the target entropy parameter is used as the fourth intermediate coefficient;

[0128] The product of the fourth intermediate coefficient and the entropy response parameter is determined as the fifth intermediate coefficient;

[0129] The activation result of the fifth intermediate coefficient is determined as the first entropy feature coefficient.

[0130] Specifically, in the above description of the target entropy parameter... The entropy response parameter is described as follows: In this case, the third objective activation function mentioned above can be expressed as:

[0131]

[0132] At the same time, it can be simplified to describe as:

[0133]

[0134] in, That is, the Sigmoid activation function. This is the entropy response coefficient (the default value can be 5). The target entropy (the default value is 70% of the uniform distribution entropy).

[0135] Correspondingly, the fourth objective activation function mentioned above can be expressed as:

[0136]

[0137] At the same time, it can be simplified to describe as:

[0138]

[0139] Having determined the first and second target activation functions, and with the similarity weights represented as follows: In this case, the stage adaptation weights corresponding to the current sample It can be determined in the following way:

[0140]

[0141] in, The policy entropy for the current training step ( (Target model strategy).

[0142] In the above embodiments, In cases of overexploration, the Sigmoid function output approaches 1, at which point the weight calculation... The proportion of (similar data) has increased significantly. In other words, when the policy is too random (high entropy), samples with a distribution similar to the test set are sampled first. The stable gradient of the known distribution is used to suppress training oscillations and prevent policy collapse.

[0143] exist In cases of overexploration / underexploration, the Sigmoid function output approaches 0, at which point the weights shift towards... (Discrepancy data) offset. In other words, when the strategy converges prematurely to a deterministic choice (low entropy), the sampling weights of samples that differ significantly from the test set are forcibly increased. By introducing out-of-distribution data, the exploration capability of the strategy is restored, avoiding getting trapped in local optima.

[0144] In the above implementation, by monitoring the decision entropy value of the policy network in real time, the sample sampling tendency is dynamically adjusted to achieve an adaptive balance between exploration and exploitation: when the policy entropy value is higher than the target entropy, the weights of samples similar to the test distribution are automatically increased, and stable gradients are used to suppress training oscillations caused by overexploration; when the policy entropy value is lower than the target entropy, the weights of dissimilar samples are automatically increased, and the policy exploration capability is restored by introducing out-of-distribution data, preventing premature convergence to a local optimum. This mechanism controls the smoothness of weight switching through the entropy response coefficient, forming a state-aware closed-loop control.

[0145] After determining the target sample loss value through the above implementation method, the training process of the current model is further explained below. Step S208-3 above, after determining the target sample loss value based on the product of the target sample weight and the loss value corresponding to the current sample, further includes:

[0146] S302, obtain the target sample loss value corresponding to each of the multiple training samples that match the current training stage;

[0147] S304. Based on the summation of the target sample loss values ​​corresponding to each of the multiple training samples, and the policy entropy that matches the current model, determine the training loss value that matches the current training stage.

[0148] S306, if the training loss value meets the target condition, the current model is determined as the target model for completing the training.

[0149] It should be noted that, in the embodiments of this application, the loss value corresponding to the current sample can be specifically a clip loss, which can be determined in the following way:

[0150] S402, Calculate the policy ratio, where the policy ratio is the ratio of the probability of the current model to the probability of the historical model to output actions under the same state;

[0151] S404, apply amplitude constraints to the policy ratio based on a preset pruning threshold to obtain the pruned policy ratio;

[0152] S406. Based on the policy ratio, the pruned policy ratio, and the dominance function, determine the loss value corresponding to the current sample, where the loss value is the smaller of the product of the policy ratio and the dominance function, and the product of the pruned policy ratio and the dominance function.

[0153] In this embodiment of the application, the loss value corresponding to the previous sample can be expressed as: Let be the clip loss for the i data points in the t-th iteration. The clip loss is used to limit the policy update magnitude.

[0154]

[0155] in, Let be the policy ratio at step t. For the dominant function, This is the clip threshold.

[0156] It should be noted that, The definition is the current strategy to be updated. Compared to the old strategy before the update In state Down Output Action The probability ratio, i.e. This ratio quantifies the degree of deviation between the old and new strategies: This indicates that the strategy has not changed. This indicates that the new strategy favors this action. This indicates that the new strategy is more effective at suppressing the action.

[0157] The advantage function is used to indicate the estimated advantage function value for the i-th sample (state-action pair), and is used to measure the execution of the action. The degree of superiority or inferiority of the action relative to the average value of that state. A positive value indicates that the action is better than the average, and a negative value indicates that it is worse than the average. It is usually calculated using the generalized advantage estimation (GAE) or Monte Carlo method.

[0158] Used to indicate the clipping operation, i.e., used to adjust the strategy ratio. Constraints in the interval [ The truncation function within ] . When Beyond the upper limit Time to take Below the lower limit Time to take Otherwise, retain the original value. This operation forcibly limits the maximum magnitude of policy updates.

[0159] Furthermore, the training loss value corresponding to the current training phase. It can be determined in the following way:

[0160]

[0161] in, For entropy regularization, Entropy weights are used to ensure the ability to explore strategies.

[0162] The target sample weights determined by the above implementation method are used to achieve domain-adaptive training guidance by weighting the loss of different samples.

[0163] Furthermore, after the current model has completed training, it can be used to perform different tasks. Optionally, after determining the current model as the target model after training, provided that the training loss value meets the target condition, it also includes at least one of the following:

[0164] Method 1: Obtain the multimodal information to be processed, which includes text information; input the multimodal information into the target model to obtain the classification result that matches the multimodal information;

[0165] Method 2: Obtain the multimodal information to be processed, which includes text information; input the multimodal information into the target model to obtain the text description result that matches the multimodal information.

[0166] In the embodiments of this application, the model training scheme based on the aforementioned proximal policy optimization (PPO) algorithm and domain adaptation mechanism can be deployed to perform inference applications in multimodal task scenarios after the current model has completed training and met the convergence conditions.

[0167] Method 1 can be applied to classification scenarios. In this embodiment, in computer vision or cross-modal understanding tasks, multimodal input information containing text, images, or sensor data (e.g., camera images and navigation text instructions in autonomous driving scenarios, or pathological images and clinical description text in medical scenarios) is acquired and input into the target model that has been trained and converged through the aforementioned adaptive weighting. Based on the domain-invariant feature representation established during training (learned through edge distribution similarity constraints and sample-level domain adaptation mechanisms), the model outputs the classification result corresponding to the input information (e.g., image category label, driving intention category, or disease diagnosis category). Because the stage-adaptive weights and entropy-aware weights effectively alleviate the domain shift between the training set (e.g., simulation data) and the test set (e.g., real-world scene data) during training, this model exhibits superior cross-domain generalization ability and classification accuracy compared to traditional supervised learning models in real-world deployment environments.

[0168] Method two can be applied to generating application scenarios. In this embodiment, in the vision-language generation task, multimodal information containing image-text pairs (such as environmental images perceived by the robot's vision and task instruction text, or mixed image-text queries in a general domain) is acquired and input into a target model optimized by difficulty-adaptive weights and a course learning mechanism. This model utilizes the complex pattern generation capabilities learned through a dynamically adjusted sampling strategy during training to output text descriptions that match the multimodal input (such as detailed descriptions of images, visual question-and-answer responses, or robot action execution instructions). Thanks to the exploration-exploitation balance maintained through clip loss and policy entropy regularization during training, this model can maintain semantic coherence and factual accuracy when generating long text descriptions. At the same time, through a domain adaptation mechanism, it ensures that the generated content conforms to the data distribution characteristics of the target domain (such as a specific professional field or a real physical environment), avoiding the generation of hallucination content that is inconsistent with the target scenario.

[0169] Through the above-described implementation methods of this application, both application methods rely on the robust strategies obtained from the aforementioned adaptive training framework of fusion stage adaptation, difficulty awareness, and entropy awareness, and achieve effective transfer from simulation / general domain training to real / specific domain deployment.

[0170] According to one aspect of this application, a model training method is also provided, such as Figure 3 As shown, it includes the following steps:

[0171] S502, determine the target sample loss value corresponding to each of the multiple training samples in the training sample set that are respectively matched with the various server operating states. The training samples include multiple operating index parameter values ​​that match the server operating states. The target sample loss value is determined based on the similarity weight matched with the training sample and the loss value corresponding to the training sample. The similarity weight is determined based on the sample similarity between the training sample and at least one reference sample in the test sample set.

[0172] S504. Based on the summation of the target sample loss values ​​corresponding to multiple training samples and the policy entropy matching the current model, determine the training loss value matching the current training stage.

[0173] S506: If the training loss value meets the target condition, the current model is determined as the target model for completing the training.

[0174] Specifically, step S502 above, which determines the target sample loss value corresponding to each of the multiple training samples in the training sample set that matches each of the various server operating states, includes:

[0175] S502-1, Obtain training samples from the training sample set;

[0176] S502-2, determine the sample similarity between the training sample and at least one reference sample in the test sample set respectively, and obtain the marginal distribution similarity between the training sample set and the test sample set, wherein the marginal distribution similarity is determined based on the difference in probability distribution between the training sample set and the test sample set in at least one feature dimension;

[0177] S502-3, determine the similarity weight matching the current sample based on at least one sample similarity and marginal distribution similarity;

[0178] S502-4, Determine the target sample loss value based on the similarity weight and the loss value corresponding to the current sample.

[0179] The following combination Figure 4 A complete model training method of this application is described.

[0180] In this embodiment, step S602 is executed first to initialize the model parameters θ. This involves randomly initializing the model parameters θ or initializing them based on pre-trained weights to provide an initial state for subsequent iterative training. Furthermore, hyperparameters related to the PPO algorithm, such as learning rate, batch size, and weighting coefficients, can also be initialized.

[0181] Next, S604, the preprocessing stage, is executed. This includes S604-1, calculating the edge similarity between the training and test sets; S604-2, calculating the text similarity between the training and test sets; and S604-3, calculating the difficulty of the training data. Multiple operational metric values ​​in the samples can include time-series values ​​(CPU utilization, memory usage, disk IOPS, network throughput, number of processes, etc., changing over time) under different server operating states, semantic and physical status indicators based on log analysis (server rack temperature, intake / exhaust air temperature, humidity, fan speed, power consumption, etc.), and information obtained from the analysis of collected images / videos (server indicator light status, hard drive activity light flashing frequency, and physical displacement / obstruction of equipment captured by data center inspection cameras).

[0182] That is to say, in the embodiments of the present application, the similarity and difficulty coefficient in the above embodiments can be determined in the preprocessing stage. In the preprocessing stage, as in S604-1, the marginal distribution similarity between the training set and the test set can be calculated based on the CORAL (correlation alignment) or MMD (maximum mean discrepancy) method to quantify the overall distribution difference between the two domains in the feature space; in S604-2, the text similarity between the training set and the test set is calculated (such as using Jaccard similarity) to capture the inter-domain association at the semantic level; in S604-3, the difficulty coefficient of each training sample is determined by using a reference inference model (such as a small pre-trained model) to statistically calculate the correct rate through multiple inferences.

[0183] Next, execute the iterative training step S606 to enter the iterative training; which further includes a judgment step S606-1 to judge whether the number of training steps < T and the performance of the validation set has not converged? If the judgment is yes, execute S606-2 to calculate the adaptive weight; the adaptive weight calculation step includes S606-21 to calculate the stage adaptive weight; S606-22 to calculate the difficulty adaptive weight; S606-23 to calculate the entropy-aware weight.

[0184] It can be understood that inside the training loop, first execute S606-2 (calculate the adaptive weight), this step calculates three types of complementary weight coefficients in parallel. S606-21 calculates the stage adaptive weight according to the normalized training progress, and realizes the smooth transition from different data to similar data through the Sigmoid function; S606-22 combines the temperature coefficient and the sample difficulty to calculate the difficulty adaptive weight to achieve the progressive difficulty of curriculum learning; S606-23 calculates the entropy-aware weight based on the deviation between the current policy entropy value and the target entropy to dynamically adjust the balance between exploration and exploitation.

[0185] Among them, the above stage adaptive weight can be determined by the foregoing embodiments, the difficulty adaptive weight can be determined according to the manner corresponding to the above second reference weight, and the entropy-aware weight can be determined according to the manner corresponding to the above third reference weight.

[0186] S606-3, combine the weights to obtain the final weighted coefficient; in this embodiment, the above stage adaptive weight, difficulty adaptive weight and entropy-aware weight can be fused to determine the final weighted coefficient of the current sample. :

[0187]

[0188] Among them, , . is the stage adaptive weight, is the difficulty adaptive weight, is the entropy-aware weight.

[0189] S606-4, Calculate the weighted loss and update the model parameters θ using SGD; continue to step S606-1 loop.

[0190] If the condition in S606-1 is not met, then S608 is executed, and the trained domain adaptive multimodal large model is output.

[0191] In the above implementation, a weighted loss function is calculated based on the weighting coefficient, and the stochastic gradient descent (SGD) algorithm is used to perform gradient updates on the model parameters θ to complete the current iteration.

[0192] When the training meets the termination condition of S606-1, the process enters S608, outputting the trained domain-adaptive multimodal large model, which has the ability to effectively generalize from the training domain to the test domain.

[0193] Through the above-described implementation method of this application, the cross-domain generalization performance of the model is improved by designing a process involving data preprocessing to calculate difficulty and similarity, weighting coefficient adjustment, and reinforcement learning optimization. The overall process and the core functions of each module are as follows: First, the preprocessing stage completes the multi-dimensional quantification of data distribution similarity and the assessment of training data difficulty, providing quantified parameters for subsequent weighting strategies; then, in the weighting coefficient calculation stage, the three key dimensions of training progress, data difficulty, and policy entropy are integrated to generate dynamically adapted sample weighting coefficients, realizing a training process from exploration to convergence and from easy to difficult; then, the weighting coefficients are integrated into reinforcement learning training, and a weighted loss function is constructed based on the PPO algorithm to guide the model to focus on learning high-value samples and improve domain adaptability; finally, the model training effect is verified through convergence judgment criteria to ensure that the model's generalization performance and training stability meet the standards, forming a complete training process, and achieving the domain-adaptive training goal through multi-dimensional collaborative adjustment.

[0194] The following preprocessing procedure is further explained. In an optional implementation, the above-described determination of sample similarity between the current sample and at least one reference sample in the test sample set includes at least one of the following:

[0195] Method 1: When the current sample includes text characters, determine the edit similarity between the current sample and at least one reference sample in the test sample set, wherein the edit similarity is determined based on the edit distance between the first text character included in the current sample and the second text character included in the reference sample;

[0196] Specifically, the above-mentioned edit similarity can quantify the similarity at the character level of text based on edit distance, and is suitable for precise matching of short texts. Given the first text character... With the second text character Edit similarity can be determined in the following ways:

[0197] ,

[0198] in, For strings and Edit distance, These represent the lengths and ranges of the two strings, respectively. The higher the value, the higher the similarity.

[0199] Method 2: When the current sample includes text characters, determine the cosine similarity between the current sample and at least one reference sample in the test sample set. The cosine similarity is determined based on the cosine distance between the first character vector of the third text character included in the current sample and the second character vector of the fourth text character included in the reference sample.

[0200] Cosine similarity quantifies semantic similarity based on the angle between word vectors and is suitable for high-dimensional text vectors. Given a third text character... With the fourth text character Cosine similarity can be determined in the following way:

[0201]

[0202] in, For the third and fourth text characters Word embedding vectors, for Norm, range of values After taking the absolute value, map to Used for similarity assessment.

[0203] Method 3: If the current sample includes text characters, determine the first set and the second set based on the fourth text character included in the current sample. The first set is the intersection between the fourth text character and the reference character set corresponding to at least one reference sample, and the second set is the union between the fourth text character and the reference character set. Determine the sample similarity based on the first set and the second set.

[0204] In the three methods described above, Jaccard similarity can be used to describe the similarity of the samples. Specifically, the Jaccard similarity can be determined based on the proportion of the intersection and union of the text word sets. Given a fourth text character... With reference character set Jaccard similarity formula:

[0205]

[0206] in, These are the fourth text character and the reference character set. The set of words, the range of values The higher the value, the higher the keyword overlap.

[0207] In one alternative implementation, the sample similarity can be determined based on any of the methods described above;

[0208] In another alternative implementation, the sample similarity can be determined based on a combination of any two or three of the methods described above. For example, if the editing similarity is determined by method one and the cosine similarity is determined by method two, the sample similarity can be determined based on the weighted sum of the editing similarity and the cosine similarity.

[0209] The following further explains the method for determining sample similarity. In an optional implementation, before determining the similarity weight matching the current sample based on at least one sample similarity and marginal distribution similarity, one of the following is also included:

[0210] Method 1: Obtain a weighted summation of at least one sample similarity, where the weighted summation is used to determine the similarity weights;

[0211] In Method 1 above, when multiple sample similarity metrics exist, a weighted summation method can be used to determine the overall similarity. Specifically, similarities from different sources are weighted and fused using a preset fusion coefficient to obtain a reference similarity used to determine the similarity weights.

[0212] Method 2: Determine the target sample similarity with the largest value from at least one sample similarity, where the target sample similarity is used to determine the similarity weight.

[0213] In Method 2 above, when multiple sample similarity values ​​are obtained—for example, the similarity calculated between the current sample and multiple candidate reference samples in the test set, or the similarity between different modalities in a multimodal scenario—the maximum value is selected as the target sample similarity. That is, if the current sample is highly similar to any reference sample in the test set or to any feature dimension (such as only image or only text), the maximum similarity value is used to determine the similarity weight. This ensures that as long as a sample is highly similar to a key representative sample in a single dimension or to a key representative sample in the test set, it receives a higher weight, avoiding misjudgments of domain association caused by averaging multi-dimensional information and enhancing the sensitivity to identifying high-value samples.

[0214] The method for determining edge distribution similarity is further explained below. In an optional implementation, obtaining the edge distribution similarity between the training sample set and the test sample set includes at least one of the following:

[0215] Method 1: Obtain the first covariance matrix of the training sample set in the target feature space and the second covariance matrix of the test sample set in the target feature space, respectively; determine the first marginal distribution similarity based on the first and second covariance matrices.

[0216] CORAL similarity quantifies the similarity of marginal distributions by aligning the covariance matrix. Given training set domain data... and test set domain data CORAL similarity formula:

[0217]

[0218] in, ... For norm, the range of values ​​is: A larger value indicates that the marginal distribution is closer.

[0219] Method 2: Map the training and test sample sets to the regenerating kernel Hilbert space respectively; determine the second marginal distribution similarity based on the mean embedding difference between the training and test sample sets according to the mapping results.

[0220] MMD similarity quantifies marginal distribution differences by using the maximum mean difference, transforming it into a similarity metric. Given training set domain data... and test set domain data MMD similarity formula:

[0221]

[0222] in, For feature mapping function, For the regenerating nucleus Hilbert space, and Training set domain data and test set domain data Two samples. Range of values. A larger value indicates a smaller difference in marginal distribution.

[0223] In one alternative implementation, either of the two methods described above can be arbitrarily determined as the edge distribution similarity.

[0224] In another alternative implementation, the weighted sum of the two similarities can be determined as the marginal distribution similarity. :

[0225]

[0226] in, The default value for the fusion weight is 0.5.

[0227] The edge distribution similarity was determined through the above implementation method. and sample similarity In this case, similarity weights can be determined in the following way. :

[0228]

[0229] The default value is 0.5.

[0230] Through the above-described implementation methods of this application, the comprehensive similarity determined by weighted fusion can comprehensively measure the overall matching degree between the sample and the target test domain in the multi-dimensional feature space, balance the contribution of different feature modalities (such as visual feature distribution and text semantic information) to domain adaptation, and make the weight allocation take into account both edge distribution alignment and individual sample correlation, thereby more accurately reflecting the domain adaptation value of the sample relative to the test scenario. On the other hand, the target similarity is determined by the maximum similarity strategy, which is based on the "maximum matching" principle. This ensures that as long as the sample is highly similar to a representative sample in a single key dimension or to the representative sample in the test set, it can obtain a high weight, avoid misjudgment of domain correlation caused by the averaging of multi-dimensional information, and enhance the sensitivity and discrimination ability to identify high-value samples. The two strategies optimize the sample selection mechanism in the domain adaptation process from the perspectives of "comprehensive evaluation" and "key identification," respectively. The weighted summation method establishes a global domain matching evaluation index, while the maximum similarity method establishes a local key sample screening mechanism. The two strategies work together to enable the model to adaptively identify and prioritize learning samples that are consistent with the distribution of the test domain or have key representativeness during the training process. This effectively alleviates the domain offset problem between the training set and the test set and improves the cross-domain generalization ability and convergence stability of the final model.

[0231] Currently, multimodal large models have become the core carrier for the application of artificial intelligence in the industry, and are widely used in dozens of core industries such as intelligent driving, medical image diagnosis, industrial visual quality inspection, intelligent customer service, AIGC content generation, embodied intelligent robots, and financial risk control. The lightweight dynamic weighted training method proposed in this application can be directly adapted to all current mainstream multimodal large model architectures and reinforcement learning training processes. It can significantly improve the cross-domain generalization performance of the model without adding new labeled data, changing the model inference structure, or increasing inference computing power. It can cover the entire industry chain, including large model training infrastructure service providers, industry large model solution providers, terminal AI product manufacturers, and open source large model communities. The potential market covers the core links of the global artificial intelligence industry and has extremely high commercial application value and industry empowerment capabilities.

[0232] This application possesses strong engineering feasibility, primarily reflected in:

[0233] Full-stack technology compatibility: Developed based on the current mainstream global Transformer multimodal large model architecture and PPO reinforcement learning training framework, it is fully compatible with all mainstream deep learning frameworks such as PyTorch, TensorFlow, and MindSpore. It can be directly adapted to all open-source and closed-source multimodal large models such as LLaVA, Qwen-VL, Yi-VL, Gemini, and Claude. No reconstruction of the main model architecture is required. Only a preprocessing similarity calculation module and a weighted coefficient calculation module need to be added, and a small amount of modification is required to the reinforcement learning loss function. The engineering transformation cost is extremely low.

[0234] Low data and computing power threshold: No additional training data needs to be collected or labeled. The entire process can be completed using only the training set and validation set (test set) in the original training process. The similarity calculation and difficulty assessment in the preprocessing stage need to be completed in one go. The computing cost is less than 5% of the overall training computing power and will not increase the additional training cost.

[0235] Strong hyperparameter robustness: All core hyperparameters are set with experimentally verified default values ​​and are extremely robust to hyperparameter fluctuations. Stable performance improvements can be achieved without extensive parameter tuning by algorithm engineers, making engineering implementation extremely easy.

[0236] High reproducibility and verifiability: This solution has completed full principle verification and performance testing based on the open-source multimodal large model LLaVA-1.5. In core tasks such as cross-domain text and image question answering, visual reasoning, and scene understanding, the model improves the generalization accuracy in unseen target domains by 18%-32%, and the stability of the training process is improved by more than 45%. All experimental results are reproducible and have a complete foundation for engineering implementation.

[0237] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0238] Embodiments of this application also provide a device for determining model training loss. Figure 5 This is a structural block diagram of a device for determining model training loss according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0239] The acquisition unit 402 is used to acquire the current sample from the training sample set that matches each of the various server operating states respectively;

[0240] The first determining unit 404 is used to determine the sample similarity between the current sample and at least one reference sample in the test sample set, and to obtain the marginal distribution similarity between the training sample set and the test sample set. The training samples in the training sample set and the test samples in the test sample set include multiple operating index parameter values ​​that match the server operating status. The marginal distribution similarity is determined based on the probability distribution difference between the training sample set and the test sample set in at least one feature dimension.

[0241] The second determining unit 406 is used to determine the similarity weight matching the current sample based on at least one sample similarity and marginal distribution similarity.

[0242] The third determining unit 408 is used to determine the target sample loss value based on the similarity weight and the loss value corresponding to the current sample. The target sample loss value is used to update the model parameters in the current model.

[0243] Through the model training loss determination device in the embodiments of this application, the current model can adaptively identify the distribution relationship between training data and target test data when updating parameters, assign high weights to samples within the distribution to make full use of effective information, and assign low weights to samples outside the distribution to suppress domain offset interference, thereby improving the cross-domain generalization ability of the model when there are marginal distribution differences between the training set and the test set.

[0244] Optionally, the third determining unit 408 includes:

[0245] The first weight determination module is used to determine the stage adaptation weight based on the similarity weight and the current training stage of the model.

[0246] The second weight determination module is used to determine at least one reference weight and determine the target sample weight that matches the current sample based on the at least one reference weight and the stage adaptation weight.

[0247] The loss determination module is used to determine the loss value of the target sample based on the product between the target sample weight and the loss value corresponding to the current sample.

[0248] Optionally, the apparatus for determining the model training loss is used for at least one of the following: using similarity weights as first reference weights; obtaining a difficulty coefficient that matches the current sample; determining a second reference weight based on the difficulty coefficient, wherein the difficulty coefficient is used to indicate the inference difficulty corresponding to the current sample; obtaining a policy entropy that matches the current model; and determining a third reference weight based on the policy entropy, wherein the policy entropy is used to indicate the degree of dispersion of the model output of the current model.

[0249] Optionally, the device for determining the model training loss is configured to: determine the ratio between the current training step and the total training step as the stage description coefficient; process the stage description coefficient according to a first target activation function to obtain a first stage feature coefficient; determine the product between the first stage feature coefficient and the similarity weight as a first adaptation weight; process the stage description coefficient according to a second target activation function to obtain a second stage feature coefficient; determine the product between the second stage feature coefficient and the reference similarity weight as a second adaptation weight, wherein the sum between the reference similarity weight and the similarity weight is 1, and the sum between the first target activation function and the second target activation function is 1; and determine the sum of the first stage feature coefficient and the second stage feature coefficient as the stage adaptation weight.

[0250] Optionally, the device for determining the model training loss is used to: use the difference between the stage description coefficient and the offset parameter as a first intermediate coefficient; use the product between the first intermediate coefficient and the smoothing parameter as a second intermediate coefficient; and use the activation result of the second intermediate coefficient as the first stage feature coefficient.

[0251] Optionally, the device for determining the model training loss is used to: determine a temperature coefficient based on the training phase, wherein the temperature coefficient is positively correlated with the training phase; determine the product between the temperature coefficient and the difficulty coefficient as a third intermediate coefficient; and determine a second reference weight matching the third intermediate coefficient based on an exponential response function.

[0252] Optionally, the device for determining the model training loss is further configured to: repeatedly input the current sample into the reference inference model multiple times; obtain the inference success rate of the reference inference model in completing the correct inference operation; and determine the difficulty coefficient based on the inference success rate, wherein the inference success rate and the difficulty coefficient have a negative correlation.

[0253] Optionally, the device for determining the model training loss is used to: determine the ratio between the current number of training steps and the total number of training steps as the stage description coefficient; determine the product between the stage description coefficient and the temperature difference, and determine the sum of the product and the initial temperature coefficient as the temperature coefficient, wherein the temperature difference is determined based on the difference between the maximum temperature coefficient and the initial temperature coefficient.

[0254] Optionally, the device for determining the model training loss is configured to: determine the current training step as the reference stage descriptor coefficient; process the reference stage descriptor coefficient according to the third objective activation function to obtain the first entropy feature coefficient; determine the product between the first entropy feature coefficient and the similarity weight as the first entropy reference weight; process the reference stage descriptor coefficient according to the fourth objective activation function to obtain the second entropy feature coefficient; determine the product between the second entropy feature coefficient and the reference similarity weight as the second entropy reference weight, wherein the sum between the reference similarity weight and the similarity weight is 1, and the sum between the third objective activation function and the third objective activation function is 1; and determine the sum of the first entropy feature coefficient and the second entropy feature coefficient as the third reference weight.

[0255] Optionally, the device for determining the model training loss is used to: use the difference between the current policy entropy determined according to the stage description coefficient and the target entropy parameter as the fourth intermediate coefficient; use the product between the fourth intermediate coefficient and the entropy response parameter as the fifth intermediate coefficient; and use the activation result of the fifth intermediate coefficient as the first entropy feature coefficient.

[0256] Optionally, the means for determining the model training loss is used for at least one of the following: when the current sample includes text characters, determining the edit similarity between the current sample and at least one reference sample in the test sample set, wherein the edit similarity is determined based on the edit distance between a first text character included in the current sample and a second text character included in the reference sample; when the current sample includes text characters, determining the cosine similarity between the current sample and at least one reference sample in the test sample set, wherein the cosine similarity is determined based on the cosine distance between a first character vector of a third text character included in the current sample and a second character vector of a fourth text character included in the reference sample; when the current sample includes text characters, determining a first set and a second set based on the fourth text character included in the current sample, wherein the first set is the intersection between the fourth text character and the reference character set corresponding to at least one reference sample, and the second set is the union between the fourth text character and the reference character set; and determining the sample similarity based on the first set and the second set.

[0257] Optionally, the means for determining the model training loss is further configured to: obtain a weighted summation result among at least one sample similarity, wherein the weighted summation result is used to determine the similarity weight; and determine the target sample similarity with the largest value from at least one sample similarity, wherein the target sample similarity is used to determine the similarity weight.

[0258] Optionally, the means for determining the model training loss is used for at least one of the following: obtaining a first covariance matrix of the training sample set in the target feature space and a second covariance matrix of the test sample set in the target feature space, respectively; determining a first marginal distribution similarity based on the first covariance matrix and the second covariance matrix; mapping the training sample set and the test sample set to the regenerating kernel Hilbert space, respectively; determining the mean embedding difference between the training sample set and the test sample set based on the mapping result; and determining a second marginal distribution similarity based on the mean embedding difference.

[0259] Optionally, the device for determining the model training loss is further configured to: obtain the target sample loss value corresponding to each of the multiple training samples that match the current training stage; determine the training loss value that matches the current training stage based on the summation of the target sample loss values ​​corresponding to the multiple training samples and the policy entropy that matches the current model; and determine the current model as the target model for completing the training if the training loss value meets the target condition.

[0260] Optionally, the device for determining the model training loss is further configured to: calculate the policy ratio, wherein the policy ratio is the ratio of the probability of the current model and the historical model outputting actions under the same state; constrain the magnitude of the policy ratio based on a preset pruning threshold to obtain the pruned policy ratio; and determine the loss value corresponding to the current sample based on the policy ratio, the pruned policy ratio, and the dominance function, wherein the loss value is the smaller of the product of the policy ratio and the dominance function, and the product of the pruned policy ratio and the dominance function.

[0261] Optionally, the means for determining the model training loss is further configured to: acquire multimodal information to be processed, wherein the multimodal information includes text information; input the multimodal information into a target model to obtain a classification result matching the multimodal information; acquire multimodal information to be processed, wherein the multimodal information includes text information; input the multimodal information into a target model to obtain a text description result matching the multimodal information.

[0262] For a description of the features in the embodiment corresponding to the device for determining model training loss, please refer to the relevant description in the embodiment corresponding to the method for determining model training loss, which will not be repeated here.

[0263] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the method for determining model training loss.

[0264] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the method for determining model training loss when running.

[0265] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0266] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the method for determining model training loss.

[0267] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the method for determining model training loss.

[0268] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0269] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0270] The foregoing has provided a detailed description of the method for determining model training loss and the model training method provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for determining model training loss, characterized in that, include: Obtain the current sample from the training sample sets that are respectively matched with the various server operating states; The sample similarity between the current sample and at least one reference sample in the test sample set is determined, and the marginal distribution similarity between the current sample and the test sample set is obtained. The training samples in the training sample set and the test samples in the test sample set include multiple operation index parameter values ​​that match the server operation status. The marginal distribution similarity is determined based on the probability distribution difference between the training sample set and the test sample set in at least one feature dimension. A similarity weight matching the current sample is determined based on at least one of the sample similarities and the marginal distribution similarities; The ratio between the current training steps and the total training steps is determined as the stage description coefficient; the stage description coefficient is processed according to the first target activation function to obtain the first stage feature coefficient; the product between the first stage feature coefficient and the similarity weight is determined as the first adaptation weight; the stage description coefficient is processed according to the second target activation function to obtain the second stage feature coefficient; the product between the second stage feature coefficient and the reference similarity weight is determined as the second adaptation weight, wherein the sum of the reference similarity weight and the similarity weight is 1, and the sum of the first target activation function and the second target activation function is 1; the sum of the first stage feature coefficient and the second stage feature coefficient is determined as the stage adaptation weight; at least one reference weight is determined, and a target sample weight matching the current sample is determined according to at least one of the reference weights and the stage adaptation weight; a target sample loss value is determined according to the product between the target sample weight and the loss value corresponding to the current sample, wherein the target sample loss value is used to update the model parameters in the current model; After determining the current model as the target model for training completion, provided that the training loss value meets the target condition, the method further includes at least one of the following: acquiring multimodal information to be processed, wherein the multimodal information includes text information; inputting the multimodal information into the target model to obtain a classification result matching the multimodal information; acquiring multimodal information to be processed, wherein the multimodal information includes text information; inputting the multimodal information into the target model to obtain a text description result matching the multimodal information.

2. The method according to claim 1, characterized in that, Determining at least one reference weight includes at least one of the following: The similarity weight is used as the first reference weight; Obtain the difficulty coefficient that matches the current sample; determine a second reference weight based on the difficulty coefficient, wherein the difficulty coefficient is used to indicate the reasoning difficulty corresponding to the current sample; Obtain the policy entropy that matches the current model, and determine the third reference weight based on the policy entropy, wherein the policy entropy is used to indicate the degree of dispersion of the model output of the current model.

3. The method according to claim 1, characterized in that, The step of processing the stage description coefficients according to the first target activation function to obtain the first stage feature coefficients includes: The difference between the stage description coefficient and the offset parameter is used as the first intermediate coefficient; The product of the first intermediate coefficient and the smoothing parameter is determined as the second intermediate coefficient; The activation result of the second intermediate coefficient is determined as the first stage feature coefficient.

4. The method according to claim 2, characterized in that, The step of determining the second reference weight based on the difficulty coefficient includes: A temperature coefficient is determined based on the training phase, wherein the temperature coefficient is positively correlated with the training phase; The product of the temperature coefficient and the difficulty coefficient is determined as the third intermediate coefficient; The second reference weight, which matches the third intermediate coefficient, is determined based on the exponential response function.

5. The method according to claim 4, characterized in that, Before obtaining the difficulty coefficient matching the current sample, the method further includes: The current sample is repeatedly input into the reference inference model; Obtain the reasoning success rate of the reference reasoning model in completing the correct reasoning operation; The difficulty coefficient is determined based on the reasoning success rate, wherein there is a negative correlation between the reasoning success rate and the difficulty coefficient.

6. The method according to claim 4, characterized in that, Determining the temperature coefficient based on the training phase includes: The ratio between the current number of training steps and the total number of training steps is determined as the stage description coefficient; The product between the stage description coefficient and the temperature difference is determined, and the sum of the product and the initial temperature coefficient is determined as the temperature coefficient, wherein the temperature difference is determined based on the difference between the maximum temperature coefficient and the initial temperature coefficient.

7. The method according to claim 2, characterized in that, The step of obtaining the policy entropy that matches the current model and determining the third reference weight based on the policy entropy includes: The current training step number is determined as the reference stage descriptor coefficient; The reference stage description coefficients are processed according to the third objective activation function to obtain the first entropy feature coefficients; The product between the first entropy feature coefficient and the similarity weight is determined as the first entropy reference weight; The reference stage description coefficients are processed according to the fourth objective activation function to obtain the second entropy feature coefficients; The product between the second entropy feature coefficient and the reference similarity weight is determined as the second entropy reference weight, wherein the sum of the reference similarity weight and the similarity weight is 1, and the sum of the third target activation function and the third target activation function is 1; The sum of the first entropy characteristic coefficient and the second entropy stage characteristic coefficient is determined as the third reference weight.

8. The method according to claim 7, characterized in that, The step of processing the reference stage descriptive coefficients according to the third objective activation function to obtain the first entropy feature coefficients includes: The difference between the current policy entropy determined based on the stage description coefficient and the target entropy parameter is used as the fourth intermediate coefficient; The product of the fourth intermediate coefficient and the entropy response parameter is determined as the fifth intermediate coefficient; The activation result of the fifth intermediate coefficient is determined as the first entropy feature coefficient.

9. The method according to claim 1, characterized in that, Determining the sample similarity between the current sample and at least one reference sample in the test sample set includes at least one of the following: When the current sample includes text characters, the edit similarity between the current sample and at least one reference sample in the test sample set is determined, wherein the edit similarity is determined based on the edit distance between a first text character included in the current sample and a second text character included in the reference sample; When the current sample includes text characters, the cosine similarity between the current sample and at least one reference sample in the test sample set is determined, wherein the cosine similarity is determined based on the cosine distance between the first character vector of the third text character included in the current sample and the second character vector of the fourth text character included in the reference sample. If the current sample includes text characters, a first set and a second set are determined based on the fourth text character included in the current sample, wherein the first set is the intersection between the fourth text character and the reference character set corresponding to at least one of the reference samples, and the second set is the union between the fourth text character and the reference character set; the sample similarity is determined based on the first set and the second set.

10. The method according to claim 9, characterized in that, Before determining the similarity weight matching the current sample based on at least one of the sample similarities and the marginal distribution similarities, one of the following is also included: Obtain a weighted summation result between at least one of the sample similarities, wherein the weighted summation result is used to determine the similarity weight; The target sample similarity with the largest value is determined from at least one of the said sample similarities, wherein the target sample similarity is used to determine the similarity weight.

11. The method according to claim 9, characterized in that, The step of obtaining the marginal distribution similarity between the training sample set and the test sample set includes at least one of the following: Obtain the first covariance matrix of the training sample set in the target feature space and the second covariance matrix of the test sample set in the target feature space; determine the first marginal distribution similarity based on the first covariance matrix and the second covariance matrix; The training sample set and the test sample set are respectively mapped to the regenerating kernel Hilbert space; the mean embedding difference between the training sample set and the test sample set is calculated based on the mapping result; and the second marginal distribution similarity is determined based on the mean embedding difference.

12. The method according to claim 1, characterized in that, After determining the target sample loss value based on the product of the target sample weight and the loss value corresponding to the current sample, the method further includes: Obtain the target sample loss value corresponding to each of the multiple training samples that match the current training stage; Based on the summation of the target sample loss values ​​corresponding to each of the multiple training samples, and the policy entropy matching the current model, the training loss value matching the current training stage is determined. If the training loss value meets the target condition, the current model is determined as the target model for completing the training.

13. The method according to claim 12, characterized in that, Before determining the target sample loss value based on the product of the target sample weight and the loss value corresponding to the current sample, the method further includes: Calculate the strategy ratio, where the strategy ratio is the ratio of the probability of the current model to the probability of the historical model to output actions under the same state; The policy ratio is constrained based on a preset pruning threshold to obtain the pruned policy ratio. Based on the policy ratio, the pruned policy ratio, and the dominance function, the loss value corresponding to the current sample is determined, wherein the loss value is the smaller of the product of the policy ratio and the dominance function, and the product of the pruned policy ratio and the dominance function.

14. A model training method, characterized in that, include: Determining the target sample loss value for each of multiple training samples in a training sample set that matches various server operating states includes: determining the ratio between the current training step and the total training steps as a stage description coefficient; processing the stage description coefficient according to a first target activation function to obtain a first stage feature coefficient; determining the product between the first stage feature coefficient and the similarity weight as a first adaptation weight; processing the stage description coefficient according to a second target activation function to obtain a second stage feature coefficient; and determining the product between the second stage feature coefficient and the reference similarity weight as a second adaptation weight, wherein the product between the reference similarity weight and the similarity weight... The sum of the first target activation function and the second target activation function is 1; the sum of the first stage feature coefficients and the second stage feature coefficients is determined as the stage adaptation weight; at least one reference weight is determined, and a target sample weight matching the current sample is determined based on at least one of the reference weights and the stage adaptation weight; the target sample loss value is determined based on the product between the target sample weight and the loss value corresponding to the current sample, wherein the training samples include multiple operating indicator parameter values ​​matching the server operating state, and the similarity weight is determined based on the sample similarity between the training samples and at least one reference sample in the test sample set; Based on the summation of the target sample loss values ​​corresponding to each of the multiple training samples, and the policy entropy matching the current model, the training loss value matching the current training stage is determined. If the training loss value meets the target condition, the current model is determined as the target model for completing the training; After determining the current model as the target model that has completed training, provided that the training loss value satisfies the target condition, the method further includes at least one of the following: acquiring multimodal information to be processed, wherein the multimodal information includes text information; inputting the multimodal information into the target model to obtain a classification result matching the multimodal information; acquiring multimodal information to be processed, wherein the multimodal information includes text information; inputting the multimodal information into the target model to obtain a text description result matching the multimodal information.

15. The method according to claim 14, characterized in that, The determination of the target sample loss value corresponding to each of the multiple training samples in the training sample set that matches each of the various server operating states includes: The training samples are obtained from the training sample set; The sample similarity between the training sample and at least one reference sample in the test sample set is determined, and the marginal distribution similarity between the training sample set and the test sample set is obtained, wherein the marginal distribution similarity is determined based on the difference in probability distribution between the training sample set and the test sample set in at least one feature dimension; A similarity weight matching the current sample is determined based on at least one of the sample similarities and the marginal distribution similarities; The target sample loss value is determined based on the similarity weight and the loss value corresponding to the current sample.

16. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as claimed in any one of claims 1 to 13 or 14 to 15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 13 or 14 to 15.

Images