A method for analyzing model performance and distinguishing authenticity in scientific literature

Abstract

This invention discloses a method for analyzing model performance and determining authenticity in scientific and technological literature, relating to the field of intelligent paper analysis. The method includes: acquiring multimodal information from the paper; extracting parameter data of the target model and inputting it into a training budget calculation model and an inference budget calculation model to obtain the training budget and inference budget; extracting parameter data of a baseline model for comparison and obtaining its training budget and inference budget; obtaining parameters from the paper regarding the performance improvement of the target model relative to the baseline model; inputting the training budget and inference budget of the target model and the baseline model, along with the performance improvement parameters, into a comparison model; and outputting the category of the actual performance improvement reasons for the target model in the paper based on the comparison model. This invention can effectively distinguish whether the model performance improvement stems from algorithm improvement or computational power stacking; it can also identify the possibility of data fabrication through experimental instrument and data consistency comparison; and it can identify potential image forgery or beautification issues in the paper by combining image processing.

Description

Technical Field

[0001] This invention relates to the field of intelligent paper analysis, and more specifically, to a method for analyzing model performance and determining authenticity in scientific and technological literature. Background Technology

[0002] In the process of researching and disseminating scientific literature, researchers often need to verify and compare the performance and efficiency of new models and methods. With the widespread application of deep learning and multimodal methods in various fields, papers often present multimodal content such as model parameters, experimental data, image results, and relevant instrument information, which constitute important evidence for the paper's conclusions.

[0003] Many existing papers emphasize improvements in accuracy, recall, or other performance metrics of a particular model, but lack objective measurement and disclosure of the computational power consumption during the training and inference phases. In this situation, it's difficult for readers to determine whether the performance improvement stems from genuine algorithmic improvement or simply from increased computing resources. Performance gains resulting from excessive computational power often obscure the algorithm's intrinsic value, hindering the objective evaluation of research findings and potentially misleading future research directions.

[0004] Furthermore, the training and inference configuration parameters given in the papers are often scattered and lack a unified measurement system, making horizontal comparisons between different papers difficult. In addition, the experimental data used in some papers may exceed the resolution or sensitivity capabilities of the instruments, or the results of image-related experiments may have been processed with pseudo-color mapping, contrast enhancement, etc., resulting in inconsistencies with the original input domain. These situations may weaken the credibility and reproducibility of the conclusions.

[0005] This demonstrates that existing research papers still have unresolved issues in distinguishing between algorithmic improvements and computational power stacking, in matching experimental data with instrument capabilities, and in ensuring the authenticity of image results. These shortcomings not only affect the objectivity of research findings but also pose potential risks to academic exchange and technology dissemination. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method for analyzing the performance and determining the authenticity of models in scientific and technological literature, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for analyzing model performance and determining authenticity in scientific and technological literature includes the following steps:

[0009] Obtain multimodal information from the paper;

[0010] The parameter data of the target model is extracted from the multimodal information, and the parameter data is input into the training budget calculation model and the inference budget calculation model to obtain the training budget and inference budget of the target model.

[0011] The parameter data of the baseline model used for comparison is extracted from the multimodal information, and the parameter data of the baseline model is input into the training budget calculation model and the inference budget calculation model to obtain the training budget and inference budget of the baseline model.

[0012] Obtain the performance improvement parameters of the target model relative to the baseline model from the paper;

[0013] The training and inference budgets of the target model and the baseline model are input into the comparison model along with the performance improvement parameters, and the actual performance improvement reasons for the target model in the paper are output based on the comparison model.

[0014] In some embodiments, the training budget calculation model is implemented using machine learning methods, specifically including:

[0015] Based on the training configuration parameters disclosed in historical paper samples and the corresponding actual computing power consumption results, a training budget prediction dataset is constructed.

[0016] Feature extraction is performed on the training budget prediction dataset, and the features include training rounds, training data size, batch size, number of model parameters, input data dimension, and data augmentation method;

[0017] The features are trained using a regression model or a neural network to obtain a training budget prediction model.

[0018] In some embodiments, the training budget calculation model calculates the training budget value based on the following formula:

[0019] ;

[0020] in, This represents the training budget value, and `train` represents the training phase. Indicates the training round; This represents the total number of samples in the training data; Indicates batch size; This represents the computational cost of forward propagation for a single sample. This represents the combined coefficient of forward propagation, back propagation, and parameter update, and its value can range from 2.5 to 4.0, preferably 3; This indicates the magnitude of data augmentation or multi-view processing during the training phase.

[0021] In some embodiments, the inference budget calculation model is implemented using machine learning methods, specifically including:

[0022] Based on the inference configuration parameters disclosed in historical paper samples and the corresponding actual inference computation consumption results, an inference budget prediction dataset is constructed.

[0023] Feature extraction is performed on the inference budget prediction dataset, and the features include the number of model parameters, the dimension of input data, the augmentation methods during testing, and the number of model ensembles.

[0024] The features are trained using a regression model or a neural network to obtain an inference budget prediction model.

[0025] In some embodiments, the inference budget calculation model calculates the inference budget value according to the following formula:

[0026] ;

[0027] in Indicates the inference budget value, and infer indicates the inference stage; This represents the forward computation cost of the model on a single sample. Indicates the enhancement factor during testing; TTA indicates the enhancement factor during testing. Indicates the number of models used in the inference process.

[0028] In some embodiments, the comparison model calculates the gain exponent using the following formula:

[0029] ;

[0030] in, Indicates the gain index; This represents the performance ratio, which is the ratio of the target model's performance metric to the baseline model's performance metric. The training budget ratio is the ratio of the training budget for the target model to the training budget for the baseline model, where t represents the training budget. The inference budget ratio is the ratio of the inference budget of the target model to the inference budget of the baseline model, where i represents inference.

[0031] The reasons for the performance improvement are evaluated based on the range of the gain index.

[0032] when When the target model is deemed to have high value, it indicates that the target model has a significant performance improvement at the algorithm level compared to the baseline model.

[0033] when When the value is medium, it means that the target model offers little performance improvement at the algorithm level compared to the baseline model.

[0034] when If the target model is deemed low-value, it means that the target model has not improved performance at the algorithm level compared to the baseline model. If there is a performance improvement, it is due to increased computing power.

[0035] In some embodiments, the method further includes:

[0036] The experimental instruments were extracted from the paper, and the instrument parameters related to the experimental instruments were obtained, including resolution, time resolution, and sensitivity.

[0037] Extract the training data used in the target model from the paper, wherein the training data is measured by the experimental instrument;

[0038] Compare whether the training data falls within the accuracy range of the instrument parameters;

[0039] If the data falls outside the accuracy range of the instrument, a warning will be issued indicating potential data falsification.

[0040] In some embodiments, the predicted data output by the target model is further compared to see if it falls within the accuracy range of the instrument parameters. Data that falls outside the accuracy range of the instrument parameters is marked as a model extrapolation conclusion.

[0041] In some embodiments, the method further includes:

[0042] Perform coordinate transformation on the input and output images of the target model to convert them to the same pixel coordinate system;

[0043] In the pixel coordinate system, an edge set G is extracted from the input image, and a boundary set H is extracted from the output image;

[0044] For each boundary point p in the output image boundary set H, calculate its Euclidean distance to the nearest point in the input image edge set. ;

[0045] Distance threshold Below, statistics satisfy Number of boundary points And based on this, calculate the boundary edge alignment rate A:

[0046] ;

[0047] A represents the number of boundary points within the boundary set H. When A is less than a preset threshold, it is determined that the boundary of the output image is inconsistent with the edge of the input image, thus indicating that the output image may not be the output of the real model or may have been processed.

[0048] In some embodiments, the step of converting the input image and the output image to the same pixel coordinate system includes:

[0049] Select multiple corresponding calibration points on the input and output images respectively;

[0050] The output image is mapped to the coordinate space of the input image based on the calibration points, or the input image is mapped to the coordinate space of the output image.

[0051] The advantage of this invention over existing technologies lies in its ability to establish a comparative classification of the reasons for performance improvements by acquiring multimodal information from research papers and combining the training and inference budgets of the target and baseline models. This solves the problem of difficulty in distinguishing the sources of performance improvements in existing literature. This approach can link computational consumption with performance improvements while ensuring scientific evaluation, clearly determining whether model improvements truly stem from algorithmic innovation, thereby enhancing the fairness and objectivity of scientific research evaluation.

[0052] Regarding the calculation of training and inference budgets, this invention provides two implementation methods: one is to construct a prediction model using historical paper samples through machine learning methods, thereby obtaining a relatively accurate budget estimate under complex parameter combinations; the other is to estimate computational power consumption directly based on factors such as training epochs, batch size, and the number of model parameters through formulaic calculations. The former has flexibility and scalability, suitable for handling complex and diverse experimental configurations; the latter has intuitiveness and interpretability, suitable for standardized measurement. Combining the different methods enhances the applicability and robustness of the approach.

[0053] Regarding the comparison models, by introducing a gain exponent formula, the performance ratio is combined with the training and inference budget ratios to form a unified evaluation metric, and the reasons for performance improvements in different intervals are provided. This approach not only quantifies the value of the research findings but also clearly distinguishes the difference between algorithmic improvements and computational power stacking, providing researchers with a reliable quantitative standard.

[0054] Furthermore, this invention considers the capability boundaries of experimental instruments, comparing the training data in the paper with the parameter range of the experimental instruments. This allows for warnings when data exceeds the instrument's measurable range, further preventing data falsification or misrepresentation. A similar comparison is performed on the predicted output data, providing quantifiable evidence of whether the model's conclusions are reasonable extrapolations. In image-based data scenarios, this invention introduces boundary-edge consistency auditing. By converting the input and output images to the same pixel coordinate system, and then performing Euclidean distance measurement and alignment calculation, it determines whether the output image truly corresponds to the input features, effectively avoiding risks associated with post-processing or inconsistencies.

[0055] It is evident that this invention not only ensures the impartiality of scientific research results from the perspective of the relationship between computing power and performance, but also enhances the credibility and reproducibility of conclusions from the perspectives of experimental instruments and image comparison, forming a comprehensive audit and evaluation mechanism, and providing strong support for the objective analysis of scientific and technological literature. Attached Figure Description

[0056] Figure 1 This is an overall structural diagram of the invention;

[0057] Figure 2 This is a schematic diagram of the training budget calculation model of the present invention;

[0058] Figure 3 This is a schematic diagram of the reasoning budget calculation model of the present invention.

[0059] Figure 4 This is a schematic diagram illustrating the consistency between the instrument and data of this invention;

[0060] Figure 5 This is a schematic diagram illustrating the consistency judgment of images before and after in this invention. Detailed Implementation

[0061] The specific embodiments of the present invention will now be described with reference to the accompanying drawings.

[0062] This invention provides a method for analyzing model performance and determining authenticity in scientific and technological literature. By integrating multimodal information such as text descriptions, images, charts, and tabular data in the paper, it can deeply analyze the performance of the target model.

[0063] Specifically, such as Figure 1 As shown, this invention collects multimodal information from research papers, including textual descriptions in the main body, graphs of experimental results, schematic diagrams of model architecture, and parameter lists in the appendix, such as the number of model parameters, training epochs, and data scale. Through comprehensive processing of this information, the training budget and inference budget of the target model can be accurately calculated, allowing for comparison with baseline models and revealing the underlying reasons for performance improvements. Simultaneously, this method also verifies the reliability of experimental data, avoiding data manipulation or inconsistent model outputs, and ensuring the objectivity of the analytical results.

[0064] In its implementation, this invention first acquires multimodal information from the paper. This step processes the paper's PDF file using optical character recognition (OCR) technology, converting the text into searchable strings, while simultaneously using image processing algorithms to extract numerical data from charts and graphs. For example, if the paper contains a line graph showcasing model performance, edge detection algorithms such as the Canny operator can be used to identify the coordinate axes, and specific numerical points can be read through pixel value mapping. The purpose of this is to comprehensively capture the implicit information in the paper, as many key parameters are often scattered throughout charts and graphs rather than plain text, thus avoiding analytical biases caused by omissions. After acquisition, this information is preprocessed, including noise removal and unit standardization, such as unifying training epochs to integer representations or converting data scales to values ​​in millions, to ensure consistency in subsequent calculations.

[0065] In a further embodiment, parameter data of the target model is extracted from multimodal information. This parameter data covers architectural details of the model, such as the number of layers and neurons, as well as training configurations, such as training epochs, batch size, and input data dimensions. For example, assuming a paper describes a convolutional neural network model, the number of parameters can be obtained by counting the convolutional kernels and fully connected layer weights. If not explicitly listed in the paper, it can be inferred from code snippets or pseudocode, for example, by calculating the total number of parameters by parsing a model definition in PyTorch or TensorFlow format. This parameter data is then input into the training budget calculation model and the inference budget calculation model to obtain the training budget and inference budget of the target model. The training budget reflects the computational resources consumed during model training, while the inference budget focuses on the efficiency of deployment, which helps quantify whether performance improvements depend on additional computing power.

[0066] Similarly, parameter data for the baseline model used for comparison is extracted from multimodal information. The baseline model is typically a standard benchmark mentioned in the paper, such as a variant of ResNet or BERT. The parameter extraction process is the same as for the target model, for example, reading the training data size or batch size from the comparison table in the paper. If the baseline model description in the paper is incomplete, reasonable default values ​​can be added, such as the standard ResNet50 having approximately 25 million parameters, to ensure the fairness of the comparison. The parameter data of the baseline model is then input into the same training budget calculation model and inference budget calculation model to calculate its training budget and inference budget. The motivation for this comparison is that simple performance metric improvements may stem from increased computing power rather than algorithmic innovation; budget quantization can remove these interfering factors and provide a more realistic evaluation.

[0067] Next, obtain the performance improvement parameters of the target model relative to the baseline model from the paper. These parameters include the percentage improvement in accuracy, the difference in F1 score, or the factor of inference speedup, and are usually extracted from the experimental results section of the paper. For example, if the accuracy of the target model is 95% and the baseline is 90% read from a table, the improvement parameter can be calculated as 5%. If the paper uses multiple metrics, a weighted average or the main metric can be selected to reflect the overall performance change.

[0068] Furthermore, the training and inference budgets of the target model and the baseline model, along with performance improvement parameters, are input into the comparison model. Based on the output of the comparison model, the category of the actual performance improvement of the target model in the paper is determined. The comparison model achieves quantitative judgment by calculating the gain index, with the formula:

[0069] ;

[0070] in Indicates the gain index. It is the performance ratio, that is, the ratio of the performance index of the target model to the performance index of the baseline model. For example, if the accuracy of the target model is 1.05 times that of the baseline, then R=1.05. It is the training budget ratio, which is the ratio of the training budget of the target model to the training budget of the baseline model; It is the inference budget ratio, which is the ratio of the inference budget of the target model to the inference budget of the baseline model.

[0071] The use of a logarithmic form is primarily to characterize the diminishing marginal returns of training and inference budgets. Specifically, as computational resources increase, model performance improvements often do not maintain a linear growth rate. For example, when the amount of training data or the number of iterations is small, increasing computational overhead can significantly improve model performance. However, when the model is fully trained or the parameter size is already large, even with additional computational resources, the performance improvement will gradually slow down. If a linear metric is still used, the gap between budget increases and performance improvements will be exaggerated, failing to accurately reflect the relationship between input and return. By introducing a logarithmic function, the mathematical structure naturally reflects the principle of "significant early-stage investment and diminishing marginal returns in later stages," thus allowing the denominator of the gain exponent to more reasonably constrain the benefits of budget increases and avoid misjudging simply increasing computational power as a high-value improvement.

[0072] Evaluation is based on the range of values ​​for the gain index:

[0073] When FGI is greater than or equal to 1.0, it is considered high value, which means that the target model has significantly improved performance at the algorithm level, possibly through novel activation functions or attention mechanisms.

[0074] When 0.7 is less than or equal to FGI and less than 1.0, it is considered to be of medium value, indicating that the algorithm's contribution is limited and the performance improvement depends partly on optimization techniques.

[0075] When the FGI is less than 0.7, it is considered low-value, suggesting that any performance improvement is mainly due to increased computing power, such as larger GPU clusters, rather than core innovation. The motivation for this classification is to help researchers identify the true contribution of a paper and avoid being misled by superficial data.

[0076] As a special case implementation, if In other words, with the budget unchanged and no performance improvement, the calculated FGI is 0.72, which falls into the mid-value category.

[0077] like Figure 2 As shown, the training budget calculation model can be implemented using machine learning methods. First, a training budget prediction dataset is constructed based on the disclosed training configuration parameters and corresponding actual computational power consumption results from historical paper samples. These historical samples can be collected from public databases such as arXiv or Papers with Code. For example, 1000 machine learning papers can be selected, and parameters such as training epochs and data size can be extracted, labeled with the actual FLOPs or GPU hours used. After the dataset is constructed, feature extraction is performed. Features include the training epochs E, the total number of training data samples N, the batch size B, the number of model parameters, the input data dimension such as image resolution, and the types and quantities of data augmentation methods such as flipping or cropping. These features are then normalized, for example, using Min-Max scaling, to improve model convergence.

[0078] Furthermore, the features are trained using a regression model or a neural network to obtain a budget prediction model. If a regression model is used, a random forest regressor can be chosen, whose architecture includes multiple decision trees, each with a depth of 10-15, and a total of 100 trees. Overfitting is reduced through bootstrap sampling and random feature selection. The training process involves dividing the dataset into an 80% training set and a 20% validation set, using mean squared error as the loss function, and using grid search for parameter tuning of the optimizer. If a neural network is used, a multilayer perceptron (MLP) can be constructed, with the number of nodes in the input layer equal to the feature dimension (e.g., 6), two hidden layers each with 128 nodes, using the ReLU activation function, and a single node in the output layer outputting the budget value. During training, the Adam optimizer is used with a learning rate of 0.001, a batch size of 32, and 100 epochs of training. The validation loss is monitored to stop early and prevent overfitting. After model training, accuracy can be evaluated through cross-validation; for example, an R² score greater than 0.9 indicates reliability. The advantage of this machine learning implementation is its ability to handle non-linear relationships, such as the complex impact of data augmentation on the budget, thus achieving higher accuracy than simple formulas.

[0079] The training budget calculation model can also calculate the training budget value based on the formula:

[0080] ;

[0081] in The training budget is represented by E, usually in FLOPs; E is the number of training epochs; N is the total number of training data samples; B is the batch size; F is the computational cost of forward propagation per sample, which can be calculated by kernel size and number of channels for convolutional layers; κ is the combined coefficient of forward propagation, backpropagation, and parameter update, usually set to 3, because backpropagation is approximately twice as computationally expensive as forward propagation; α is the data augmentation or multi-view processing factor during training, for example, if two augmentations are used, then α=2. The motivation behind this formula is to decompose the computational components of the training process, making it easier to estimate directly from the parameters in the paper. If some parameters are missing in the paper, they can be supplemented with domain averages, such as F for computer vision tasks, which defaults to 1e9 FLOPs per sample.

[0082] like Figure 3 As shown, the inference budget calculation model can also be implemented using machine learning methods. First, an inference budget prediction dataset is constructed based on the disclosed inference configuration parameters and corresponding actual inference computation costs from historical paper samples. For example, model parameter counts, input data dimensions, the number of boosting techniques such as TTA during testing, and the number of model ensembles such as ensemble size are extracted from 500 papers, and actual inference FLOPs are labeled. Feature extraction includes the quantization of these parameters; for example, the input dimension is represented by vector length, and boosting techniques are encoded using multiples. The features are trained using a regression model or neural network to obtain the inference budget prediction model. The regression model can use gradient boosting trees, such as XGBoost, with parameters including a learning rate of 0.1, a tree depth of 6, and 200 trees, with complexity controlled by a regularization term. The training process is similar, involving dataset splitting, using mean squared error loss, and cross-validation. The neural network architecture can be an MLP, with 4 nodes in the input layer corresponding to features, two hidden layers each with 64 nodes, ReLU activation, and output budget values. Training uses the SGD optimizer with a momentum of 0.9, 50 epochs, and 16 batches. This method can capture the multiplicative effect of the ensemble model on the budget, thus improving prediction accuracy.

[0083] The inference budget calculation model can also calculate the inference budget value according to the formula:

[0084] ;

[0085] in Indicates the inference budget value; It represents the forward computation cost of the model on a single sample; β is the augmentation factor during testing; for example, 3 is the augmentation factor if there is one flip and one rotation. β is the number of models used in the inference ensemble; for example, 5 is the number of models in the ensemble. This formula is simple and easy to estimate quickly. The motivation is to highlight the additional overhead of the inference stage, such as augmentations used to improve robustness.

[0086] In a further embodiment, the method of the present invention also extracts experimental instruments from the paper and obtains instrument parameters related to the experimental instruments, including resolution such as pixel density, temporal resolution such as frame rate, and sensitivity such as signal-to-noise ratio. For example, if the paper involves an optical microscope, its resolution of 0.1 micrometers can be extracted. Training data related to the target model used in the paper, measured by the experimental instruments, such as image datasets, is extracted. Then, the training data is compared to see if it falls within the accuracy range of the instrument parameters; for example, if the data claims an accuracy of 0.01 micrometers but the instrument limit is 0.1, it is out of range. If it falls outside the instrument accuracy range, a warning is given that data fabrication may exist. The motivation for this verification is to ensure the reliability of the data source, as fabricated data may lead to inflated model performance.

[0087] like Figure 4 As shown, the method of the present invention further compares whether the predicted data output by the target model falls within the accuracy range of the instrument parameters. Data falling outside the accuracy range of the instrument parameters is marked as a model extrapolation conclusion. For example, if the physical quantity predicted by the model exceeds the upper limit of the instrument's measurement, it indicates that the conclusion is an inference and may be difficult to verify through actual measurement, reminding the user of potential uncertainties. This step enhances the reliability of the method and avoids blindly trusting the model output.

[0088] like Figure 5 As shown, in a further embodiment, the present invention also performs coordinate transformation on the input and output images of the target model, converting them to the same pixel coordinate system. This is achieved by selecting multiple corresponding calibration points on the input and output images, such as image corner points or significant features like intersecting lines, and then mapping the output image to the coordinate space of the input image based on the calibration points, or conversely, using an affine transformation matrix to calculate, for example, by solving the transformation parameters using the least squares method, to ensure that the pixel alignment error is less than 1 pixel.

[0089] In pixel coordinates, an edge set G is extracted from the input image. The Sobel operator is used to detect gradients and threshold the edges to obtain the edge point set. Similarly, a boundary set H is extracted from the output image, and edge detection or thresholding is used to obtain boundary points. For each boundary point p in the output image boundary set H, the Euclidean distance dist(p,G) to the nearest point in the input image edge set is calculated, and the nearest neighbor search is accelerated using a KD-tree. At a distance threshold δ (e.g., δ=5 pixels), the number of boundary points satisfying dist(p,G)<δ is counted as |{p∈H:dist(p,G)<δ}|, and the boundary edge alignment rate A=(|{p∈H:dist(p,G)<δ}|) / |H| is calculated accordingly, where |H| is the number of boundary points in the boundary set H. When A is less than a preset threshold, such as 0.8, the boundary of the output image is determined to be inconsistent with the edge of the input image, indicating that the output image may not be the output of a real model or may have been post-processed, such as edited in Photoshop. The motivation for this check is to detect image forgery, because the output of a real model should retain the geometric consistency of the input.

[0090] In summary, this invention not only quantifies the reasons for the performance improvement but also verifies the authenticity of the data and the reliability of the output. In practical applications, it can be extended to papers in different fields, such as computer vision or natural language processing, adjusting feature extraction to adapt to specific modalities. For example, in NLP, adding token length as a feature can further improve the comprehensiveness of the parsing.

[0091] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for analyzing model performance and determining authenticity in scientific and technological literature, characterized in that, Includes the following steps: Obtain multimodal information from the paper; the multimodal information includes textual descriptions in the main body of the paper, charts of experimental results, schematic diagrams of model architecture, and parameter lists in the appendix; The parameter data of the target model is extracted from the multimodal information, and the parameter data is input into the training budget calculation model and the inference budget calculation model to obtain the training budget and inference budget of the target model; the parameter data covers the architectural details of the target model, including the number of layers, the number of neurons, and the training configuration; The parameter data of the baseline model used for comparison is extracted from the multimodal information, and the parameter data of the baseline model is input into the training budget calculation model and the inference budget calculation model to obtain the training budget and inference budget of the baseline model. Obtain the performance improvement parameters of the target model relative to the baseline model from the paper; The performance improvement parameters include percentage improvement in accuracy, difference in F1 score, or acceleration of inference speed. The training and inference budgets of the target model and the baseline model are input into the comparison model along with the performance improvement parameters. Based on the comparison model, the actual performance improvement reasons for the target model in the paper are output. The comparison model uses the following formula to calculate the gain index: ； in, Indicates the gain index; This represents the performance ratio, which is the ratio of the target model's performance metric to the baseline model's performance metric. This represents the training budget ratio, which is the ratio of the training budget for the target model to the training budget for the baseline model. This represents the inference budget ratio, which is the ratio of the inference budget of the target model to the inference budget of the baseline model. The reasons for the performance improvement are evaluated based on the range of the gain index. when When the target model is deemed to have high value, it indicates that the target model has a significant performance improvement at the algorithm level compared to the baseline model. when When the value is medium, it means that the target model offers little performance improvement at the algorithm level compared to the baseline model. when If the target model is deemed low-value, it means that the target model has not improved performance at the algorithm level compared to the baseline model. If there is a performance improvement, it is due to increased computing power.

2. The method according to claim 1, characterized in that, The training budget calculation model is implemented using machine learning methods, specifically including: Based on the training configuration parameters disclosed in historical paper samples and the corresponding actual computing power consumption results, a training budget prediction dataset is constructed. Feature extraction is performed on the training budget prediction dataset, and the features include training rounds, training data size, batch size, number of model parameters, input data dimension, and data augmentation method; The features are trained using a regression model or a neural network to obtain a training budget prediction model.

3. The method according to claim 1, characterized in that, The training budget calculation model calculates the training budget value based on the following formula: ； in, This represents the training budget value; Indicates the training round; This represents the total number of samples in the training data; Indicates batch size; This represents the computational cost of forward propagation for a single sample. This represents the combined coefficient of forward propagation, back propagation, and parameter update. This indicates the magnitude of data augmentation or multi-view processing during the training phase.

4. The method according to claim 1, characterized in that, The inference budget calculation model is implemented using machine learning methods, specifically including: Based on the inference configuration parameters disclosed in historical paper samples and the corresponding actual inference computation consumption results, an inference budget prediction dataset is constructed. Feature extraction is performed on the inference budget prediction dataset, and the features include the number of model parameters, the dimension of input data, the augmentation methods during testing, and the number of model ensembles. The features are trained using a regression model or a neural network to obtain an inference budget prediction model.

5. The method according to claim 1, characterized in that, The inference budget calculation model calculates the inference budget value according to the following formula: ； in Indicates the inference budget value; This represents the forward computation cost of the model on a single sample. Indicates the enhancement factor during testing; Indicates the number of models used in the inference process.

6. The method according to claim 1, characterized in that, The method further includes: The experimental instruments were extracted from the paper, and the instrument parameters related to the experimental instruments were obtained, including resolution, time resolution, and sensitivity. Extract the training data used in the target model from the paper, wherein the training data is measured by the experimental instrument; Compare whether the training data falls within the accuracy range of the instrument parameters; If the data falls outside the accuracy range of the instrument, a warning will be issued indicating potential data falsification.

7. The method according to claim 6, characterized in that, The method further includes: comparing whether the predicted data output by the target model falls within the accuracy range of the instrument parameters, and marking the data that falls outside the accuracy range of the instrument parameters as model extrapolation conclusions.

8. The method according to claim 1, characterized in that, The method further includes: Perform coordinate transformation on the input and output images of the target model to convert them to the same pixel coordinate system; In the pixel coordinate system, an edge set G is extracted from the input image, and a boundary set H is extracted from the output image; For each boundary point p in the output image boundary set H, calculate its Euclidean distance to the nearest point in the input image edge set. ; Distance threshold Below, statistics satisfy Number of boundary points And based on this, calculate the boundary edge alignment rate A: ; A represents the number of boundary points within the boundary set H. When A is less than a preset threshold, it is determined that the boundary of the output image is inconsistent with the edge of the input image, thus indicating that the output image may not be the output of the real model or may have been processed.

9. The method according to claim 8, characterized in that, The steps to convert the input and output images to the same pixel coordinate system include: Select multiple corresponding calibration points on the input and output images respectively; The output image is mapped to the coordinate space of the input image based on the calibration points, or the input image is mapped to the coordinate space of the output image.

Images