A method and apparatus for intelligent evaluation of functional and performance differences in information technology application innovation (ITAI) transformation systems.
By synchronously collecting and preprocessing multimodal data, and combining semantic embedding models and temporal decomposition models, the functional and performance differences of the domestic IT innovation transformation system are identified. A comprehensive evaluation report is generated through association rule mining, which solves the problem of insufficient joint collection and intelligent association analysis of functional and performance data in the verification of domestic IT innovation transformation, and realizes efficient and accurate intelligent evaluation.
Patent Information
- Application Number
- CN202611222374.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-12
- Publication Date
- 2026-11-10
AI Technical Summary
Existing verification methods for information technology innovation transformation lack joint collection and intelligent correlation analysis of functional and performance data, resulting in incomplete functional verification coverage, low efficiency, and inability to reveal deep correlations, thus affecting the level of automation, standardization, and intelligence.
Multimodal data is collected and preprocessed synchronously. Pre-trained semantic embedding models and structured difference classifiers are used to identify functional differences. Performance bias is analyzed by combining time-series decomposition models. Association rules are established to establish association rules between function and performance, and a comprehensive evaluation report is generated.
It enables automatic identification and correlation of functional and performance differences in IT innovation transformation systems, improves the accuracy and efficiency of evaluation, provides intelligent root cause localization and comprehensive quantitative evaluation, and supports quality access control for DevOps pipelines.
Smart Images

Figure CN123070116A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to an intelligent evaluation method and apparatus for the functional and performance differences of a domestically developed information technology innovation system. Background Technology
[0002] As a core component of the national information security strategy, the "Information Technology Application Innovation" (ITAI) initiative is driving the migration of core business systems in critical industries such as finance and telecommunications from traditional x86 architectures to a full-stack ITAI environment featuring domestically produced CPUs, operating systems, and databases. Among related technologies, a basic framework for verifying the functionality and performance of systems after ITAI transformation has been constructed through collaborative operations using methods such as manual regression testing with predefined test cases, traffic replay based on recording / playback technology, and monitoring and comparison based on log keywords or rule thresholds. Specifically, this verification system covers the entire process from executing functional test cases and replaying production traffic to collecting performance indicators, including key aspects such as request-response comparison, TPS and response time statistics, and anomaly log filtering.
[0003] However, existing verification methods for domestic IT innovation transformation directly employ the traditional approach of comparing independent tests with manual reports. This approach fails to achieve joint collection and intelligent correlation analysis of functional and performance data. Consequently, functional verification may be incomplete and inefficient (e.g., semantic equivalence noise drowns out real differences), or performance and functionality may be analyzed in isolation, failing to reveal deep correlations (e.g., the inability to automatically determine whether performance degradation stems from logical errors or bottlenecks). This impacts the automation, standardization, and intelligence levels of domestic IT innovation transformation verification, making it difficult to form a quality gate that can be embedded in DevOps pipelines. Summary of the Invention
[0004] The present invention aims to at least partially solve one of the technical problems in the related art.
[0005] Therefore, the first objective of this invention is to propose an intelligent evaluation method for the functional and performance differences of information technology innovation transformation systems.
[0006] Another objective of this invention is to propose an intelligent evaluation device for the functional and performance differences of a domestic information technology innovation transformation system.
[0007] The third objective of this invention is to provide a computer device.
[0008] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.
[0009] To achieve the above objectives, a first aspect of the present invention proposes an intelligent evaluation method for the functional and performance differences of a system undergoing domestic information technology innovation transformation, comprising: S10: Under the same or equivalent load conditions, collect transaction log data and performance index time series data of the system before and after the information technology innovation transformation, and perform preprocessing operations such as time alignment, transaction alignment and structured parsing on the collected data. S20 uses a pre-trained semantic embedding model to encode the response messages of the systems before and after the modification into semantic vectors. By calculating semantic similarity and combining it with a structured difference classifier, it automatically identifies and distinguishes semantically equivalent response differences from real business logic differences, and outputs a functional difference report. S30: The performance index time series data of the system before the transformation is sliced according to transaction type. The normal behavior baseline of each index is established by using time series decomposition or time series prediction model. Based on the baseline, the abnormal score of the corresponding time series data of the system after the transformation is calculated, the time period and type of performance deviation are identified, and a performance difference report is output. S40: Associate the actual logical difference events in the functional difference report with the performance anomaly events in the performance difference report within a time window. Use an association rule mining algorithm to generate association rules between functional differences and performance anomalies. Generate a comprehensive evaluation report based on the association rules, functional equivalence score, and performance stability score.
[0010] In one embodiment of the present invention, S10 includes: The time-series data of the performance indicators of the system before and after the transformation are joined by timestamps, and the missing seconds are filled by linear interpolation to obtain a time-aligned performance indicator sequence. Using a globally unique transaction identifier as the primary key, request-response pairs with the same transaction identifier in the systems before and after the modification are associated to form a triple containing the request identifier, the response before the modification, and the response after the modification. Transactions that cannot be matched are marked as orphan transactions. Structured request and response messages are parsed into dictionary objects. If the original message is in binary format, it is decoded according to the protocol to obtain structured dictionary data. Outliers in the performance index time series data are identified using the 3σ criterion and replaced with the mean of the preceding and following time points to obtain a smoothed performance index sequence.
[0011] In one embodiment of the present invention, S20 includes: The pre-trained financial domain semantic embedding model is used to encode the pre-modification response and the post-modification response into high-dimensional semantic vectors respectively; Calculate the cosine similarity of two semantic vectors. If the similarity is higher than 0.95 and both business return codes are success, then mark them as semantically equivalent differences and ignore them. If the similarity is less than 0.95, the structured difference classifier based on XGBoost is used to determine whether the difference type is a real business logic difference or a configurable difference based on the characteristics of whether the return code is consistent, the difference of the numerical field, and the existence of the key field. The function difference report containing the real logic difference points and their related transaction identifiers is then output.
[0012] In one embodiment of the present invention, S30 includes: The performance index time series data of the system before the transformation is sliced according to transaction type. For each slice, the trend term, cycle term and residual term are decomposed using STL decomposition or Prophet model to establish a normal behavior baseline model. For the corresponding time-series data of the modified system, calculate the anomaly score at each time point: , in To ensure the modified system at a specific point in time The index value, and The baseline model at time points are respectively The mean and standard deviation; Time periods with outlier scores exceeding 2-3σ are marked as significant shifts, and the shift type is identified as mean shift, variance increase, or abrupt change. A performance difference report containing the shift time period, relevant indicators, and shift type is output.
[0013] In one embodiment of the present invention, S40 includes: The performance anomaly events within a 30-second time window before and after the time point of each real logical difference event are correlated to form a transaction dataset. The FP-Growth or Apriori algorithm is used to mine association rules across all transaction sessions. The rules are in the form of functional difference type to performance anomaly type, and support and confidence are calculated. Output the most reliable association rules as root cause localization clues.
[0014] In one embodiment of the present invention, it further includes: Calculate the functional equivalence score = (1 - number of transactions with actual logical differences / total number of transactions) × 100; The performance stability score is calculated as 100 - the normalized average anomaly score, where the normalized average anomaly score is mapped using the sigmoid function. The overall score is calculated as follows: α × functional equivalence score + β × performance stability score, where α and β are configurable weights.
[0015] In one embodiment of the present invention, it further includes: Using template generation technology, functional difference reports, performance difference reports, association rules, functional equivalence scores, performance stability scores, and comprehensive scores are filled into a pre-set report template; Generate a comprehensive assessment report that includes natural language diagnostic text.
[0016] To achieve the above objectives, a second aspect of the present invention provides an intelligent evaluation device for the functional and performance differences of a domestically developed information technology innovation system, comprising: The data preprocessing module is used to collect transaction log data and performance index time series data of the system before and after the information technology innovation transformation under the same or equivalent load conditions, and to perform preprocessing operations such as time alignment, transaction alignment and structured parsing on the collected data. The functional difference detection module is used to encode the response messages of the system before and after the modification into semantic vectors using a pre-trained semantic embedding model. By calculating semantic similarity and combining it with a structured difference classifier, it automatically identifies and distinguishes semantically equivalent response differences from real business logic differences and outputs a functional difference report. The performance deviation analysis module is used to slice the time series data of the performance indicators of the system before the transformation according to the transaction type, establish the normal behavior baseline of each indicator using time series decomposition or time series prediction model, calculate the abnormal score of the corresponding time series data of the system after the transformation based on the baseline, identify the time period and type of performance deviation, and output a performance difference report. The comprehensive correlation assessment module is used to correlate the actual logical difference events in the functional difference report with the performance anomaly events in the performance difference report within a time window. It uses a correlation rule mining algorithm to generate correlation rules between functional differences and performance anomalies, and generates a comprehensive assessment report based on the correlation rules, functional equivalence score, and performance stability score.
[0017] The present invention discloses an intelligent evaluation method and apparatus for functional and performance differences of a system undergoing information technology innovation transformation. This method and apparatus can automatically identify the functional differences and performance deviations of the system before and after the transformation, establish the correlation between the two, and achieve intelligent root cause localization and comprehensive quantitative evaluation.
[0018] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing the method described in the first aspect embodiment.
[0019] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.
[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0021] Figure 1 This is a flowchart of an intelligent evaluation method for functional and performance differences in a domestically developed information technology innovation system according to an embodiment of the present invention; Figure 2 This is a structural diagram of an intelligent evaluation device for functional and performance differences in a domestic information technology innovation transformation system according to an embodiment of the present invention; Figure 3 It is a computer device according to an embodiment of the present invention. Detailed Implementation
[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0024] The following describes, with reference to the accompanying drawings, an intelligent evaluation method and apparatus for the functional and performance differences of an information technology innovation transformation system proposed according to an embodiment of the present invention.
[0025] Figure 1 This is a flowchart of an intelligent evaluation method for functional and performance differences in a domestically developed information technology innovation (ITAI) system, according to an embodiment of the present invention. Figure 1 As shown, it includes: S10: Under the same or equivalent load conditions, collect transaction log data and performance index time series data of the system before and after the information technology innovation transformation, and perform preprocessing operations such as time alignment, transaction alignment and structured parsing on the collected data. Furthermore, under the same or equivalent load conditions, multimodal data were synchronously collected from both the baseline system before the IT innovation transformation and the target system after the transformation. The collected data included at least transaction log data reflecting the system's functional logic and time-series performance index data reflecting the system's operating status. The transaction log data covered the request and response information for each transaction, while the performance index time-series data recorded key operating parameters of the system at the application layer, system layer, and middleware or database layer in a time series format. To ensure the comparability and accuracy of subsequent analysis, preprocessing operations were performed on the collected raw data. These preprocessing operations included at least time alignment, transaction alignment, and structured parsing. Time alignment was used to match the performance index data of the baseline system and the target system according to a unified time scale to eliminate deviations caused by asynchronous collection times. Transaction alignment was used to associate transaction request and response pairs with the same globally unique identifier in the baseline system and the target system, thereby forming directly comparable data pairs. Structured parsing was used to convert the raw transaction messages or log data into a standardized structured format suitable for subsequent analysis and processing. As one implementation method, during off-peak periods, real transaction requests from the production environment can be copied by bypassing to simultaneously run and collect data from both systems, or a unified automated test script can be used to perform load testing in a test environment with the same hardware specifications. Performance indicators can be continuously collected at the second level using monitoring tools. In the preprocessing stage, missing data can be filled by interpolation, and outlier values can be smoothed.
[0026] It is understandable that the synchronous acquisition and preprocessing of the multimodal data can provide a high-quality, aligned and comparable data foundation for subsequent functional difference identification and performance deviation analysis, thereby effectively eliminating noise and bias introduced by inconsistent data acquisition or non-uniform formats, and significantly improving the accuracy and reliability of the evaluation results.
[0027] S20 uses a pre-trained semantic embedding model to encode the response messages of the systems before and after the modification into semantic vectors. By calculating semantic similarity and combining it with a structured difference classifier, it automatically identifies and distinguishes semantically equivalent response differences from real business logic differences, and outputs a functional difference report. Furthermore, in the process of evaluating the functional differences between the systems before and after the IT innovation transformation, by mapping the response messages returned by the systems before and after the transformation for the same transaction request to a high-dimensional semantic space and calculating their semantic similarity in that space, it is possible to automatically identify response pairs that are semantically equivalent but may have character-level differences. Based on this, by further combining the difference analysis of the structured features of the response messages, a pre-trained classifier is used to discriminate response pairs with semantic similarity below a preset threshold, thereby distinguishing between semantically equivalent differences caused by changes in fields without business impact (such as timestamps, serial numbers, format differences, etc.) and real business logic differences caused by changes or errors in business logic. Through the above processing, a functional difference report containing all real logical differences and their associated transaction identifiers can be automatically generated. As one implementation method, a pre-trained financial domain semantic embedding model (such as Fine-tuned RoBERTa or SimCSE) can be used to encode the response message into a semantic vector, and semantic similarity can be calculated by cosine similarity. At the same time, a structured difference classifier based on XGBoost can be used to identify the difference type by taking features such as return code consistency, numerical field difference and key field existence as input.
[0028] Understandably, this step, by combining deep semantic understanding with precise analysis of structured features, enables automated and high-precision identification of functional differences. It can effectively filter out a large number of noise differences that have no business impact, significantly reduce the workload of manual analysis, and ensure that real business logic differences are accurately captured, providing a reliable functional difference data foundation for subsequent performance correlation analysis and comprehensive evaluation.
[0029] S30: The performance index time series data of the system before the transformation is sliced according to transaction type. The normal behavior baseline of each index is established by using time series decomposition or time series prediction model. Based on the baseline, the abnormal score of the corresponding time series data of the system after the transformation is calculated, the time period and type of performance deviation are identified, and a performance difference report is output. Furthermore, in the performance difference detection step, the collected time-series data of the performance indicators of the pre-modification system are first sliced according to transaction type to ensure the comparability of performance data under the same type of transaction. For each performance indicator, the historical time-series data of the pre-modification system is analyzed using a time-series decomposition or time-series prediction model to extract its inherent trend, periodicity, and randomness components, thereby constructing a behavioral baseline model for that indicator under normal operating conditions. This baseline model can dynamically reflect the normal fluctuation range and statistical distribution characteristics of the system at different time scales. Subsequently, the time-series data of the performance indicators of the post-modification system under the corresponding transaction type are input into the baseline model. By comparing the actual observed values with the predicted values or statistical distribution parameters of the baseline model, the anomaly score at each time point is calculated. The anomaly score can be calculated based on residual statistics, probability density estimation, or distance metrics. For example, one implementation uses a formula... To quantify the degree of deviation, among which To ensure the modified system is in constant motion The observed values, and These represent the mean and standard deviation of the baseline model at that time. Based on whether the outlier scores exceed a preset statistical significance threshold (e.g., 2 or 3 times the standard deviation), the system automatically identifies time periods where significant performance shifts occur and further determines the type of shift, including mean shift, increased variance, abrupt trend changes, or periodic disruptions. Finally, the detected performance shift information is summarized to generate a performance difference report containing details such as the time period of the shift, the relevant metrics, the type of shift, and its severity.
[0030] Understandably, this step, by establishing a dynamic baseline model to replace the traditional static threshold, achieves adaptive and intelligent detection of performance deviations. It can effectively capture the minor and continuous performance degradation introduced by the domestic IT innovation transformation, avoid false alarms caused by business peak fluctuations or environmental noise, and significantly improve the accuracy and robustness of performance difference assessment.
[0031] S40: Associate the actual logical difference events in the functional difference report with the performance anomaly events in the performance difference report within a time window. Use an association rule mining algorithm to generate association rules between functional differences and performance anomalies. Generate a comprehensive evaluation report based on the association rules, functional equivalence score, and performance stability score.
[0032] Furthermore, the actual logical difference events identified in the functional difference report and the performance anomaly events detected in the performance difference report are correlated within a preset time window to establish a potential causal relationship between them. Specifically, a transaction dataset for association rule mining is constructed by aligning abnormal transaction records at the functional level with abnormal indicator periods at the performance level along the time dimension. Based on this, an association rule mining algorithm is used to automatically discover frequent itemsets and strong association rules between functional difference types and performance anomaly types from this dataset. These rules can reveal the statistical correlation between code logic changes in specific functional modules and the degradation of specific performance indicators. For example, as an implementation, association rules of the form "functional difference type = misalignment of amount field in transfer module" and "performance anomaly type = database CPU spike" can be mined on all transaction sessions based on FP-Growth or Apriori algorithms, and their support and confidence scores are calculated to evaluate the reliability of the rules. Subsequently, a comprehensive evaluation report is generated based on the mined association rules, combined with the functional equivalence score calculated based on the ratio of the number of functional differences to the total number of transactions, and the performance stability score obtained by normalizing the performance anomaly score. This report integrates evaluation results from both functional and performance dimensions and provides recommendations for root cause analysis, thus offering a comprehensive and quantitative basis for decision-making in the verification of systems after domestic IT innovation transformation.
[0033] Understandably, by correlating functional differences with performance anomalies, this step can automatically establish a causal relationship between functional logic changes and performance degradation, significantly improving the efficiency and accuracy of root cause identification and avoiding the tediousness and subjectivity of manual cross-checking in traditional methods. Simultaneously, the comprehensive evaluation report based on correlation rules and quantitative scores provides an objective basis for the standardized and automated evaluation of system transformation effects, supports embedding into continuous integration processes, and achieves an intelligent closed loop for the verification of information technology innovation transformation.
[0034] Furthermore, this invention jointly collects, intelligently analyzes, and mines the transaction logs (functional modalities) and performance time-series indicators (performance modalities) of the systems before and after the domestic IT innovation transformation, thereby automatically, accurately, and comprehensively evaluating functional equivalence and performance stability, and outputting root cause localization suggestions. This invention includes five main steps: S1: Synchronous acquisition and preprocessing of multimodal data; S2: Intelligent recognition of functional differences based on deep text similarity; S3: Performance Deviation Analysis Based on Temporal Anomaly Detection; S4: Function-Performance Difference Association Rule Mining and Root Cause Localization; S5: Comprehensive quantitative scoring and natural language report generation.
[0035] Step S1: Multimodal data synchronous acquisition and preprocessing. Under the same or equivalent load conditions (e.g., the same number of concurrent users, the same transaction mixing ratio, and the same stress test time window), data is collected from both the system before the IT innovation transformation (referred to as System A, the baseline system) and the system after the transformation (referred to as System B, the target system). Two acquisition methods are recommended: Production mirror traffic acquisition: During off-peak periods, real transaction requests are copied in a bypass manner, and both systems are run simultaneously to ensure that the input load is completely consistent. Automated regression stress testing: Using a unified test script, it is executed sequentially or in parallel on a test environment with the same hardware specifications.
[0036] Furthermore, for functional modal data, request-response logs for each transaction are collected, including at least the following fields: globally unique transaction identifier (such as UUID or business transaction number), request timestamp, structured request message, response timestamp, structured response message, business return code (such as "0000" for success, "1001" for insufficient balance), and transaction type (such as "transfer", "withdrawal", "balance query"). After collection, the data is stored as log files or message queues for System A and System B respectively.
[0037] Furthermore, performance modality data involves collecting time-series data of key performance indicators during system runtime, continuously collected at the second-level granularity, including at least: Application layer indicators: average response time, maximum response time, transactions per second (TPS), success rate. System layer indicators: CPU utilization (overall and per core), memory utilization, disk I / O wait time. Middleware / database indicators: active database connection pool, average database execution time, JVM heap memory utilization, GC pause time. Collection tools can include Prometheus + NodeExporter + JMXExporter, or a customized agent. The data format is a sequence of triplets: timestamp-metric name-metric value.
[0038] Further, data preprocessing involves performing the following cleaning and alignment operations on the raw data: Time alignment: Performance metrics of system A and system B are joined by timestamps to ensure data comparability within the same second. Missing seconds are filled using linear interpolation. Transaction alignment: Using a globally unique transaction identifier as the primary key, request-response pairs with the same transaction ID in system A and system B are associated to form triples.<Req_ID,Res_A,Res_B> For transactions that cannot be matched (existing only in one system), they are marked as "orphan transactions". Structured parsing: Structured request and response messages are parsed into dictionary objects for easy field-level operations. If the original message is binary, it is first decoded according to the protocol (e.g., Protobuf, Hessian). Outlier smoothing: Obvious outliers in performance metrics (e.g., transiently high response times due to network jitter) are identified using the 3σ criterion and replaced with the mean of the preceding and following times to avoid interfering with baseline modeling.
[0039] Step S2: Intelligent identification of functional differences based on deep text similarity. Feature extraction: Res_A and Res_B are encoded into high-dimensional semantic vectors using a pre-trained financial domain semantic embedding model (such as Fine-tuned RoBERTa or SimCSE). Semantic equivalence judgment: The cosine similarity between the two response vectors is calculated, and a high threshold (e.g., 0.95) is set. If the similarity is higher than the threshold and both return codes are successful, they are marked as "semantically equivalent," and such noise differences are ignored. Real logical difference detection: If the similarity is lower than the threshold, a structured difference detector (such as an XGBoost-based classifier, with features including: whether the return codes are consistent, differences in numerical fields, and differences in the existence of key fields) is further used to determine whether it is a "real business logic difference" (such as incorrect balance calculation or reversed transfer status) or a "configurable difference" (such as date format or error code text). Output a functional difference report, listing all real logical difference points and their associated transaction IDs.
[0040] Step S3: Based on performance offset analysis using time-series anomaly detection, the time-series data of performance indicators are sliced according to transaction type (e.g., withdrawal, transfer, query) to ensure comparability among similar transactions. For each performance indicator's time-series data in System A, the STL decomposition Prophet model is used to decompose it into trend, period, and residual terms, establishing a "normal behavior baseline model" for that indicator. For the corresponding time-series data in System B, the anomaly score at each time point is calculated: Further examples of formulas include: `Score(t)=(X_B(t)-μ_baseline(t)) / σ_baseline(t)` (based on residual statistics) or using Isolation Forest to perform unsupervised anomaly detection on the data points of system B. The output is a performance difference report, marking the time periods and related metrics where performance exhibits "significant shifts" (anomaly scores exceeding 2-3σ), and indicating the type of shift (mean shift, increased variance, abrupt change, etc.).
[0041] Step S4: Association rule mining and root cause localization for functional-performance differences. This involves associating performance anomalies within the time window (30 seconds before and after) where the "real logical differences" marked in Step S2 occurred. Using FP-Growth or Apriori algorithms, association rules are mined across all trading sessions, in the following form: Further, a rule example: {Functional Difference Type = Misalignment of Amount Field in Transfer Module} -> {Performance Anomaly Type = Database CPU Surge} (Support = 0.05, Confidence = 0.85). The mining dimensions include the transaction type where the difference occurred, the specific component causing the difference (such as database, cache, message queue), and performance anomaly indicators. The output correlation report lists the most reliable rules, directly indicating to the development team that "a code change in a certain functional module is likely causing a certain type of performance problem."
[0042] Step S5: Comprehensive evaluation and report generation, quantitative scoring, and weighted calculation of the comprehensive score (0-100 points) based on the following dimensions: Functional equivalence score = (1 - number of transactions with actual logical differences / total number of transactions) × 100; Performance stability score = 100 - normalized value of average anomaly score (mapped using sigmoid function); Comprehensive score = α Functional Score +β Performance score (α and β are configurable, e.g., 0.5 each); Natural language report generation: Using template-based generation technology, the above-mentioned differences, key association rules, and quantitative scores are automatically filled into a pre-set report template. For example: "This evaluation completed the comparison of 100,000 transactions and found three types of real functional differences, concentrated in the 'interbank transfer' module. At the same time, a continuous abnormal offset in the number of database connections under this module was detected (an average increase of 40%), and the confidence level of the association rules reached 92%. It is recommended to prioritize checking the database connection pool configuration and transaction commit logic in the domestic IT innovation transformation."
[0043] The embodiments of the present invention also have the following technical effects: Manual analysis of log or message differences is easily overwhelmed by "semantic equivalence noise" such as timestamps and IDs, resulting in low efficiency. Automated identification of semantic equivalence reports only true logical differences, offering accuracy, efficiency, and 100% coverage (all traffic / all use cases). Through association rule mining, it automatically outputs quantifiable rules for "functional difference → performance anomaly," significantly reducing location time. Quantitative comprehensive scoring combined with structured natural language reports provides objective, standardized, and integrable automated access control.
[0044] To achieve the above embodiments, such as Figure 2 As shown, this embodiment also provides an intelligent evaluation device 10 for the functional and performance differences of a domestically developed information technology innovation (ITAI) system, comprising: The data preprocessing module 100 is used to collect transaction log data and performance index time series data of the system before and after the information technology innovation transformation under the same or equivalent load conditions, and to perform preprocessing operations such as time alignment, transaction alignment and structured parsing on the collected data. The functional difference detection module 200 is used to encode the response messages of the system before and after the modification into semantic vectors using a pre-trained semantic embedding model. By calculating semantic similarity and combining it with a structured difference classifier, it automatically identifies and distinguishes semantically equivalent response differences from real business logic differences and outputs a functional difference report. The performance deviation analysis module 300 is used to slice the time series data of the performance indicators of the system before the transformation according to the transaction type, establish the normal behavior baseline of each indicator by using time series decomposition or time series prediction model, calculate the abnormal score of the corresponding time series data of the system after the transformation based on the baseline, identify the time period and type of performance deviation, and output a performance difference report. The comprehensive correlation evaluation module 400 is used to correlate the real logical difference events in the functional difference report with the performance anomaly events in the performance difference report within a time window. It uses an association rule mining algorithm to generate association rules between functional differences and performance anomalies, and generates a comprehensive evaluation report based on the association rules, functional equivalence score, and performance stability score.
[0045] An intelligent evaluation device for the functional and performance differences of a system undergoing information technology innovation transformation according to an embodiment of the present invention can automatically identify the functional differences and performance deviations of the system before and after the information technology innovation transformation, establish the correlation between the two, and realize intelligent root cause localization and comprehensive quantitative evaluation.
[0046] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 3 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the method described above.
[0047] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.
[0048] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0049] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
Claims
1. A method for intelligently evaluating the functional and performance differences of a system undergoing domestic IT innovation transformation, characterized in that, include: S10: Under the same or equivalent load conditions, collect transaction log data and performance index time series data of the system before and after the information technology innovation transformation, and perform preprocessing operations such as time alignment, transaction alignment and structured parsing on the collected data. S20 uses a pre-trained semantic embedding model to encode the response messages of the systems before and after the modification into semantic vectors. By calculating semantic similarity and combining it with a structured difference classifier, it automatically identifies and distinguishes semantically equivalent response differences from real business logic differences, and outputs a functional difference report. S30: The performance index time series data of the system before the transformation is sliced according to transaction type. The normal behavior baseline of each index is established by using time series decomposition or time series prediction model. Based on the baseline, the abnormal score of the corresponding time series data of the system after the transformation is calculated, the time period and type of performance deviation are identified, and a performance difference report is output. S40: Associate the actual logical difference events in the functional difference report with the performance anomaly events in the performance difference report within a time window. Use an association rule mining algorithm to generate association rules between functional differences and performance anomalies. Generate a comprehensive evaluation report based on the association rules, functional equivalence score, and performance stability score.
2. The method as described in claim 1, characterized in that, S10 includes: The time-series data of the performance indicators of the system before and after the transformation are joined by timestamps, and the missing seconds are filled by linear interpolation to obtain a time-aligned performance indicator sequence. Using a globally unique transaction identifier as the primary key, request-response pairs with the same transaction identifier in the systems before and after the modification are associated to form a triple containing the request identifier, the response before the modification, and the response after the modification. Transactions that cannot be matched are marked as orphan transactions. Structured request and response messages are parsed into dictionary objects. If the original message is in binary format, it is decoded according to the protocol to obtain structured dictionary data. Outliers in the performance index time series data are identified using the 3σ criterion and replaced with the mean of the preceding and following time points to obtain a smoothed performance index sequence.
3. The method as described in claim 1, characterized in that, S20 includes: The pre-trained financial domain semantic embedding model is used to encode the pre-modification response and the post-modification response into high-dimensional semantic vectors, respectively. Calculate the cosine similarity of two semantic vectors. If the similarity is higher than 0.95 and both return codes are success, then mark them as semantically equivalent differences and ignore them. If the similarity is less than 0.95, the structured difference classifier based on XGBoost is used to determine whether the difference type is a real business logic difference or a configurable difference based on the characteristics of whether the return code is consistent, the difference of the numerical field, and the existence of the key field. The function difference report containing the real logic difference points and their related transaction identifiers is then output.
4. The method as described in claim 1, characterized in that, S30 includes: The performance index time series data of the system before the transformation is sliced according to transaction type. For each slice, the trend term, cycle term and residual term are decomposed using STL decomposition or Prophet model to establish a normal behavior baseline model. For the corresponding time-series data of the modified system, calculate the anomaly score at each time point: , in To ensure the modified system at a specific point in time The index value, and The baseline model at time points are respectively The mean and standard deviation; Time periods with outlier scores exceeding 2-3σ are marked as significant shifts, and the shift type is identified as mean shift, variance increase, or abrupt change. A performance difference report containing the shift time period, relevant indicators, and shift type is output.
5. The method as described in claim 1, characterized in that, S40 includes: The performance anomaly events within a 30-second time window before and after the time point of each real logical difference event are correlated to form a transaction dataset. The FP-Growth or Apriori algorithm is used to mine association rules across all transaction sessions. The rules are in the form of functional difference type to performance anomaly type, and support and confidence are calculated. Output the most reliable association rules as root cause localization clues.
6. The method as described in claim 5, characterized in that, The method further includes: Calculate the functional equivalence score = (1 - number of transactions with actual logical differences / total number of transactions) × 100; The performance stability score is calculated as 100 - the normalized average anomaly score, where the normalized average anomaly score is mapped using the sigmoid function. The overall score is calculated as follows: α × functional equivalence score + β × performance stability score, where α and β are configurable weights.
7. The method as described in claim 6, characterized in that, The method further includes: Using template generation technology, functional difference reports, performance difference reports, association rules, functional equivalence scores, performance stability scores, and comprehensive scores are filled into a pre-set report template; Generate a comprehensive assessment report that includes natural language diagnostic text.
8. An intelligent evaluation device for the functional and performance differences of a domestically developed information technology innovation (ITAI) system, characterized in that, include: The data preprocessing module is used to collect transaction log data and performance index time series data of the system before and after the information technology innovation transformation under the same or equivalent load conditions, and to perform preprocessing operations such as time alignment, transaction alignment and structured parsing on the collected data. The functional difference detection module is used to encode the response messages of the system before and after the modification into semantic vectors using a pre-trained semantic embedding model. By calculating semantic similarity and combining it with a structured difference classifier, it automatically identifies and distinguishes semantically equivalent response differences from real business logic differences and outputs a functional difference report. The performance deviation analysis module is used to slice the time series data of the performance indicators of the system before the transformation according to the transaction type, establish the normal behavior baseline of each indicator using time series decomposition or time series prediction model, calculate the abnormal score of the corresponding time series data of the system after the transformation based on the baseline, identify the time period and type of performance deviation, and output a performance difference report. The comprehensive correlation assessment module is used to correlate the actual logical difference events in the functional difference report with the performance anomaly events in the performance difference report within a time window. It uses a correlation rule mining algorithm to generate correlation rules between functional differences and performance anomalies, and generates a comprehensive assessment report based on the correlation rules, functional equivalence score, and performance stability score.
9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.
