Software operation informatization evaluation method and system based on artificial intelligence

Abstract

The invention relates to the related technical field of digital data processing, in particular to a software operation informatization evaluation method and system based on artificial intelligence, and the method comprises the steps: determining a software quality standard; effective data is obtained in a random evaluation scene, a first evaluation result is obtained through analysis of a database intelligent module, a second evaluation result is obtained through evaluation of software self-adaption and risk confrontation ability, and a target evaluation result is obtained through mean value calculation after verification of the first evaluation result and the second evaluation result. The technical problem that software operation informatization evaluation results are local and one-sided is solved, effective scene operation data are utilized, evaluation deviation caused by limitation of test samples is avoided, software operation data are deeply mined and analyzed, an elastic relaxation coefficient identifier is determined, and the adaptation flexibility of software under changing environments and demand conditions is ensured. And the basic operation performance of the software and the self-optimization and potential threat resistance capabilities in a continuously changing environment are considered, and the technical effects of completeness of an evaluation level and result accuracy are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field related to digital data processing, and in particular to an artificial intelligence-based software operation information evaluation method and system. Background Art

[0002] Artificial intelligence technology has made significant progress, especially in the fields of machine learning, natural language processing, computer vision, etc., which has promoted the digital transformation of various industries and provided powerful tools and technical support for software quality assessment.

[0003] With the widespread application of software in various industries, especially the increasingly prominent core role in key business areas, the demand for software operation quality assessment is becoming increasingly stringent. It not only requires the assessment to cover multiple dimensions such as functionality, performance, security, and compatibility, but also emphasizes the in-depth investigation of the software's adaptability, stability, and synergy in the actual operating environment. The existing software operation informationization assessment methods focus on the single attribute or local function of the software, lack of overall assessment of the software operation, and especially ignore the software's adaptive ability in complex environments and its interactive performance with other software.

[0004] In summary, the existing technology has the technical problems of single indicator for software quality assessment standard and partial and one-sided evaluation results of software operation informatization. Summary of the invention

[0005] This application provides an artificial intelligence-based software operation informationization evaluation method and system, aiming to solve the technical problems in the prior art of single indicator of software quality evaluation standards and partial and one-sided software operation informationization evaluation results.

[0006] In view of the above problems, the technical solution to implement this application is:

[0007] In one aspect, the present application provides a software operation information evaluation method based on artificial intelligence, wherein the method comprises: determining the quality standard of the target software, wherein the quality standard comprises a rigid quality standard and a flexible quality standard, and the quality measurement dimension comprises a single evaluation dimension and an interactive evaluation dimension;

[0008] Determine an evaluation scenario based on the target software, and determine valid scenario operation data, wherein the evaluation scenario is at least two random ones;

[0009] Connecting to a software database, taking the quality standard as a benchmark, performing an evaluation analysis based on the evaluation scenario in an intelligent evaluation module, including obtaining a homography matrix and evaluating the indicators based on the scenario interval, and determining a first evaluation result;

[0010] In combination with the software database, the target software is evaluated for its operation adaptive adjustment capability and risk resistance capability to determine a second evaluation result, wherein the second evaluation result is marked with an elastic relaxation coefficient;

[0011] Verifying the first evaluation result and the second evaluation result, including independent verification and mutual verification, to determine a verification result;

[0012] If the verification result is qualified, the first evaluation result and the second evaluation result are averaged and used as the target evaluation result.

[0013] On the other hand, the present application provides a software operation information evaluation system based on artificial intelligence, wherein the system includes: a standard determination module, which is used to determine the quality standard of the target software, wherein the quality standard includes a rigid quality standard and a flexible quality standard, and the quality measurement dimension includes a single evaluation dimension and an interactive evaluation dimension;

[0014] An operation data determination module, used to determine an evaluation scenario based on the target software and determine effective scenario operation data, wherein the evaluation scenario is at least two random ones;

[0015] An evaluation and analysis module, which is used to connect to a software database, and to perform evaluation and analysis based on the evaluation scenario in an intelligent evaluation module based on the quality standard, including homography matrix acquisition and index interval evaluation based on the scenario interval, to determine a first evaluation result;

[0016] A capability assessment module, used to evaluate the operation adaptive adjustment capability and risk resistance capability of the target software in combination with the software database, and determine a second assessment result, wherein the second assessment result is marked with an elastic relaxation coefficient;

[0017] A verification module, used to verify the first evaluation result and the second evaluation result, including independent verification and mutual verification, to determine a verification result;

[0018] The mean calculation module is used to calculate the mean of the first evaluation result and the second evaluation result as the target evaluation result if the verification result is qualified.

[0019] In summary, one or more technical solutions provided in this application solve the technical problems of a single indicator in the software quality assessment standard and a partial and one-sided evaluation result of the software operation informationization, realize the use of effective scenario operation data, avoid evaluation bias caused by test sample limitations, and on the basis of real-time and objective software evaluation, determine the identifier with elastic relaxation coefficient through in-depth mining and analysis of software operation data to ensure the adaptability of the software under changing environments and demand conditions, take into account the basic operating performance of the software and its ability to self-optimize and resist potential threats in a constantly changing environment, conduct a comprehensive analysis based on diversified dimensions, adaptability and scale randomness, and ensure the completeness of the evaluation level and the accuracy of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a software operation information evaluation method based on artificial intelligence is provided for this application;

[0021] Figure 2 A flowchart of a qualified verification result in an artificial intelligence-based software operation information evaluation method is provided for this application;

[0022] Figure 3 A structural diagram of an artificial intelligence-based software operation information evaluation system is provided for this application.

[0023] Explanation of the reference numerals: standard determination module M100, operation data determination module M200, evaluation and analysis module M300, capability evaluation module M400, verification module M500, mean calculation module M600. DETAILED DESCRIPTION

[0024] Embodiment 1

[0025] The present application is described in detail below with reference to the accompanying drawings. Figure 1 As shown, the present application provides a software operation information evaluation method based on artificial intelligence, wherein the method comprises:

[0026] S1: Determine the quality standard of the target software, wherein the quality standard includes a rigid quality standard and a flexible quality standard, and the quality measurement dimension includes a single evaluation dimension and an interactive evaluation dimension;

[0027] S2: Determine an evaluation scenario based on the target software, and determine valid scenario operation data, wherein the evaluation scenario is at least two random ones;

[0028] Existing software operation information evaluation methods focus on a single attribute or local function of the software. Furthermore, based on preset and limited test scenarios, it is difficult to simulate the complexity and diversity of the real world, resulting in the evaluation results being unable to accurately reflect the performance of the software in actual operation. In addition, there are deficiencies in identifying potential risks of the software and evaluating its risk resistance capabilities. Remediation is only carried out after the problem occurs, rather than early warning and proactive response.

[0029] Based on this, this application has built a comprehensive, real-time, accurate, and automated software operation information evaluation system. It has been verified that this application takes into account the basic operating performance of the software and its ability to self-optimize and resist potential threats in a constantly changing environment, effectively responding to the complex evaluation needs in modern software development and operation environments, and improving the scientificity and effectiveness of software quality management;

[0030] Specifically, the quality standards include rigid quality standards, wherein the rigid quality standards include: functionality (the target software meets all predetermined functional requirements, including core functions, auxiliary functions and interface compatibility with external systems), reliability (assessing the target software's ability to complete expected functions within specified conditions and time, including indicators such as trouble-free working time, MTTR (Mean time to repair), and failure rate), performance efficiency (measuring the performance of the target software in processing requests, response time, resource utilization (CPU, memory, disk, network), etc.); security (reviewing the target software's compliance and protection capabilities in data protection, access control, encryption mechanism, vulnerability prevention, etc.), and compatibility (checking the adaptability of the target software in different operating systems, hardware configurations, browser versions, third-party components, etc.);

[0031] The quality standards include flexible quality standards, wherein the flexible quality standards include: maintainability (evaluating the clarity of the target software architecture, code readability, modularity, documentation completeness, and the difficulty of error diagnosis and repair), scalability (inspecting the target software's ability to easily add new functions, support more users, and handle larger loads in the face of future demand changes), flexibility (measuring the target software's adaptability to changes in business rules, user interfaces, data formats, etc., and the richness of configuration options), user-friendliness (evaluating the target software's ease of use, user experience (UX / UI design), and effectiveness of help documents and training materials), and portability (evaluating the ease of migration of the target software between different environments or platforms, including the conversion costs of code, data, and configuration files);

[0032] The quality measurement dimensions include single-item evaluation dimensions, wherein the single-item evaluation dimensions include: functional testing (designing and executing test cases to verify the correctness and completeness of various functions of the target software), performance testing (simulating stress scenarios such as high concurrency and large data volumes to measure response time and resource consumption), stability testing (running the target software for a long time to monitor whether there are problems such as memory leaks and resource exhaustion, as well as system recovery capabilities), security testing (using penetration testing, vulnerability scanning and other means to check whether the target software has security vulnerabilities and verify the effectiveness of mechanisms such as access control and data encryption), and compatibility testing (running the target software under a combination of multiple operating systems, browsers and devices to ensure normal cross-platform operation);

[0033] The quality measurement dimension includes an interaction evaluation dimension, wherein the interaction evaluation dimension refers to the operation response status of the target software when working in collaboration with other software, and the interaction evaluation dimension includes: collaborative software identification (determining other software systems that work closely with the target software, such as database management systems, middleware, third-party APIs, etc.), interaction feature analysis (sorting out key interaction characteristics such as interface protocols, data exchange formats, message transmission mechanisms, synchronous / asynchronous processing modes, etc. between the target software and the multiple software connected to it), and interaction performance indicators (formulating evaluation indicators for collaborative work, such as interface response time, data transmission rate, message loss rate, synchronization delay, etc.);

[0034] Determine an evaluation scenario based on the target software and collect valid scenario operation data. Specifically, create at least two random evaluation scenarios based on the actual application scenarios of the target software, covering normal operation, peak load, fault recovery, boundary conditions and other situations; prepare corresponding input data, configuration parameters, user behavior scripts, etc. for each scenario to ensure the representativeness and validity of the data; run the software in a controlled environment, execute predefined scenario operations, and record system logs, performance indicators, user feedback and other operation data; use monitoring tools, log analysis tools, user surveys and other methods to systematically collect operation data in each scenario, and record the collected operation data in each scenario as valid scenario operation data to ensure the comprehensiveness of the data; the above operations evaluate the target software with rigid quality standards and flexible quality standards, covering single evaluation dimensions and interactive evaluation dimensions, and derive comprehensive and accurate evaluation results based on valid scenario operation data.

[0035] S3: connecting to a software database, taking the quality standard as a benchmark, performing an evaluation analysis based on the evaluation scenario in an intelligent evaluation module, including obtaining a homography matrix and evaluating the indicators based on the scenario interval, and determining a first evaluation result;

[0036] S4: performing an operation adaptive adjustment capability assessment and a risk resistance capability assessment on the target software in combination with the software database to determine a second assessment result, wherein the second assessment result is marked with an elastic relaxation coefficient;

[0037] Connect to the software database. Specifically, obtain the necessary information such as the database access address, port, user name, password, database name, etc., and configure the database connection parameters; use the appropriate database driver (such as JDBC, ODBC, ADO.NET, etc.) to establish a stable connection with the software database according to the configuration information; confirm that the connecting user has the appropriate permissions to query the required data tables, views, stored procedures and other resources to ensure that data access is not restricted during the assessment process;

[0038] Import the determined rigid quality standards and flexible quality standards into the intelligent evaluation module as the reference benchmark of the evaluation algorithm; input the designed evaluation scenario information into the intelligent evaluation module, including scenario description, expected results, relevant data sources, etc.; set the working mode, data sampling frequency, result output format and other parameters of the intelligent evaluation module according to the evaluation requirements, and further extract the performance data, log records, user behavior data and other data related to the scenario from the software database according to the evaluation scenario; conduct statistical analysis on the data of different evaluation scenarios, calculate the homography matrix between the key indicators of the scenarios, and reflect the mapping relationship and change law between the indicators; based on the homography matrix, convert the original data into interval form (such as quantile, interquartile range, standardized score, etc.) to facilitate cross-scenario comparison and trend analysis; in the actual operation process, apply machine learning, data mining and other technologies, combined with quality standards, conduct in-depth analysis of the intervalized indicators, identify strengths and weaknesses, outliers and room for improvement; summarize the results of the homography matrix analysis to form a preliminary evaluation conclusion, including comparative analysis between scenarios, key indicator performance, overall quality score, etc., as the first evaluation result;

[0039] Extract data related to software running status adjustment from the software database, such as configuration change records, automatic optimization logs, performance tuning events, etc.; analyze data related to software running status adjustment, and extract evidence of the software's self-adjustment of resource allocation, algorithm parameters, cache strategy, etc. when factors such as different loads, user behaviors, and system environments change. Further, use clustering, regression, time series analysis and other methods to build a mathematical model that describes the software's adaptive adjustment capabilities, and quantify its response speed, adjustment range, effect persistence and other indicators; compare the model prediction results with the actual effects, evaluate the strength, stability and effectiveness of the software's adaptive adjustment capabilities in various scenarios, and give a score or grade;

[0040] Extract data related to security events, fault records, and anomaly detection from the software database to identify potential risks and risk events that have occurred; evaluate the impact of each risk event on software operation, taking into account factors such as interruption time, data loss, and decreased user satisfaction; examine the defense, recovery, and compensation measures taken by the software after a risk event occurs, as well as the corresponding execution effects. Furthermore, based on multiple dimensions such as the frequency of risk events, the degree of impact, and the effectiveness of response measures, use mathematical models to calculate the elastic relaxation coefficient of the software to reflect its risk resistance capabilities; integrate the evaluation results of the adaptive adjustment capability and the risk resistance capability, i.e., the second evaluation result, which is marked with an elastic relaxation coefficient; through the above steps, obtain the homography matrix, perform interval evaluation of indicators, perform adaptive adjustment capability and risk resistance capability evaluation, and generate a second evaluation result with an elastic relaxation coefficient to provide support for subsequent analysis.

[0041] S5: verifying the first evaluation result and the second evaluation result, including independent verification and mutual verification, to determine a verification result;

[0042] S6: If the verification result is qualified, the first evaluation result and the second evaluation result are averaged and used as the target evaluation result.

[0043] Independently verify the first evaluation results. Specifically, ensure that the first evaluation results cover all predetermined evaluation dimensions and indicators without omissions; compare historical data, industry standards or theoretical expectations to verify the rationality and credibility of the evaluation results; analyze the internal consistency of the homography matrix and indicator interval evaluation results. For example, the comparison results between scenarios should be consistent with the overall evaluation conclusions;

[0044] Independently verify the second assessment results. Specifically, confirm that the data used to assess the adaptive adjustment capability and risk resistance capability are derived from the software database and that the collection process is correct; review the elastic relaxation coefficient calculation process and the data used to ensure that the calculation process is correct; the results of the adaptive adjustment capability and risk resistance capability assessment should be logically consistent with the software characteristics and usage environment;

[0045] The first evaluation result and the second evaluation result are cross-checked, specifically, cross-comparison verification: compare the scenario performance in the first evaluation result with the adaptive adjustment and risk resistance performance of the software in the corresponding scenario in the second evaluation result, and determine whether there is an intrinsic connection between the two; if the weights of each part in the evaluation system are known, check whether the weight distribution of the first evaluation result and the second evaluation result complies with the preset rules and whether the weight calculation is accurate; deviation fluctuation range verification: according to the evaluation standard or experience value, set the maximum allowable deviation range between the first evaluation result and the second evaluation result; calculate the numerical difference between the first evaluation result and the second evaluation result in each common evaluation dimension; compare the calculated difference metric with the allowable deviation range to confirm that the differences in all dimensions are within the allowable range;

[0046] Summarize the problems, anomalies or confirmed correctness information found in independent verification and mutual verification; judge whether the first evaluation result and the second evaluation result meet the verification qualification limit according to the severity and number of the problems. If all the inspection items pass and there are no major problems, they are deemed qualified; then, according to the provisions of the evaluation system or the opinions of experts, assign weights to the first evaluation result and the second evaluation result (such as 50% each or other proportions); multiply the first evaluation result and the second evaluation result by their corresponding weights respectively, and then sum them to obtain the weighted average, which is the target evaluation result;

[0047] In the actual operation process, it also includes recording in detail the process, methods, problems found and solutions of independent verification and mutual verification in the evaluation report; writing the calculated target evaluation results and their calculation process into the evaluation report; adjusting the evaluation conclusions according to the target evaluation results, and proposing targeted improvement suggestions or next action plans;

[0048] Through the above steps, the first assessment result and the second assessment result are independently verified and mutually verified to ensure data accuracy. When qualified, calculations are performed to obtain the target assessment results, and the assessment report is updated. This achieves a comprehensive analysis of diversified dimensions, adaptability, and scale randomness, ensuring the completeness of the assessment level and the accuracy of the results.

[0049] Furthermore, after obtaining the target evaluation result, the present application method includes:

[0050] Conducting homology search on the target software to determine a set of similar software;

[0051] Execute inter-class function ranking and assembly calculation of the target software and the class software set to determine the class evaluation result, wherein the assembly calculation principle is a weighted calculation based on the priority of functions;

[0052] A software evaluation result is determined based on the target evaluation result and the class evaluation result.

[0053] Determine the definition criteria of homologous software corresponding to the target software, including development language, core algorithm, application field, user group, similar functional modules, etc.; use professional databases, open source project platforms, industry reports, academic papers, market analysis reports and other resources to formulate detailed search keywords, Boolean operators and screening conditions; use search engines and professional software indexing tools to find software collections with homologous characteristics to the target software according to the search strategy; conduct a preliminary screening of the search results, eliminate software that obviously does not meet the homologous standards, retain a list of suspected homologous software, and, if necessary, consult software documents, source codes or conduct trials to verify their homology; organize and confirm the final list of homologous software to form a class software set;

[0054] For each software in the target software and its similar software collection, list its main functional modules or features to form a detailed function list; for each function, assign primary and secondary weights based on user needs, industry standards, technology development trends and other factors. The weights can be determined by expert scoring, questionnaire surveys, literature research, etc.; in the similar software collection, conduct horizontal comparisons of the same or similar functions and rank them according to their importance weights in their respective software. The ranking of the same function in different software should reflect its relative value in the software ecosystem; for each software in the similar software collection, according to the function ranking results, use a weighted total calculation based on the primary and secondary functions;

[0055] Based on the target evaluation results, summarize the scores of each indicator and the analysis conclusions to form an overall evaluation profile of the target software; for each software in the class software set, combine its overall calculation score to form a class evaluation profile, including the average score, the highest score, the lowest score, functional advantages and disadvantages, etc.; compare the target software evaluation profile with the class evaluation profile to analyze the position, advantages and disadvantages, innovations, unique value, etc. of the target software in the homologous software set; based on the comparative analysis results, combine the target evaluation results with the class evaluation results, and conduct a comprehensive evaluation of the target software, including the performance of the target software relative to other software in the class software set and the applicability of the target software in specific application scenarios; through the above steps, perform homologous retrieval, inter-class functional ranking and overall calculation on the target software, and obtain a comprehensive software evaluation result after comparative analysis, which provides a scientific basis for software optimization, competitive strategy, etc.

[0056] Furthermore, the quality measurement dimension includes an interactive evaluation dimension, and the method of the present application includes:

[0057] Determine collaborative software of the target software and determine collaborative features, wherein the collaborative features are determined based on software interactive functions;

[0058] A first weight distribution is performed based on interaction dominance, and a second weight distribution is performed based on interaction relevance, wherein the relative dominance based on software interaction is marked;

[0059] Determine a multivariate evaluation index of the interactive evaluation dimension, and perform hierarchical weighting processing based on the first weight distribution and the second weight distribution.

[0060] By analyzing the functional description, interface documents, configuration files or actual usage of the target software, identify other software that works directly or indirectly with the target software. Collaborative software includes API service providers, database management systems, third-party plug-ins, hardware drivers, client applications, etc. For each collaborative software, list in detail the interaction interface between it and the target software, including API interfaces, message queues, database query statements, file transfer protocols, etc., and clarify key information such as interaction methods, data formats, and access rights;

[0061] Based on the interaction interface, summarize the collaborative features between the target software and each collaborative software, including: interaction frequency (the number of times data is exchanged or request responses are received between software), data dependency (whether one software relies on the data provided by the other to run normally), synchronous / asynchronous mode (whether real-time response is required during the interaction, or asynchronous processing is allowed), fault tolerance mechanism (how does the target software ensure service continuity when the collaborative software fails or is delayed), and security requirements (whether security measures such as data encryption, identity authentication, and access control are complete);

[0062] The first weight distribution is performed based on the interaction dominance. Specifically, the interaction dominance refers to the influence or control of the target software on other software in the collaborative work, which can be manifested in the direction of data flow, the initiative of task scheduling, the allocation of responsibility for error handling, etc. Further, for each collaborative work software, according to the interactive relationship between it and the target software, its dominance in the interactive process is evaluated. The evaluation can be quantitatively scored (such as 1 to 5 points). Then, based on the evaluation results, a first weight value is assigned to each collaborative work software. In general, the more dominant the software is, the larger the corresponding first weight should be, indicating that it has a greater impact on the interactive quality of the target software;

[0063] The second weight distribution is performed based on the interactive correlation. Specifically, the interactive correlation measures the degree of functional dependence and data coupling between the target software and the collaborative software, thereby reflecting the closeness of the interaction between the software and the risk of fault propagation. For each collaborative software, the criticality of its function in the business process of the target software, the depth and breadth of data exchange, the impact range caused by the fault and other factors are analyzed to evaluate its interactive correlation with the target software. Furthermore, based on the evaluation results, a second weight value is assigned to each collaborative software. Generally speaking, the higher the correlation degree of the software, the larger the corresponding second weight should be, indicating that it has a greater impact on the overall function and stability of the target software.

[0064] Select specific indicators suitable for evaluating the interactive performance of software, such as response time, data consistency, concurrent processing capability, fault recovery speed, interface compatibility, user experience, etc.; combine the first weight distribution with the second weight distribution to perform hierarchical weighting on the multivariate evaluation indicators, and further sort all collaborative work software according to the first weight to form a dominant weight sequence; for each software, adjust its position in the sequence according to its second weight to form a comprehensive weight sequence; then, assign corresponding weights to the multivariate evaluation indicators according to the position of the corresponding software in the comprehensive weight sequence, and the evaluation indicators corresponding to the software with higher weights should be given more attention; through the above steps, the logical decomposition of the interactive evaluation dimensions of the target software and the refinement of the implementation process are realized, ensuring the comprehensiveness, accuracy and operability of the interactive performance evaluation.

[0065] Furthermore, the intelligent evaluation module is communicatively connected with the software database, and includes a homography processing unit and an adaptive evaluation unit, wherein the adaptive evaluation unit includes a static evaluation branch and a dynamic evaluation branch, and an evaluation analysis based on the evaluation scenario is performed in the intelligent evaluation module. The method of the present application includes:

[0066] Based on the software database, extracting scenario operation information based on the evaluation scenario;

[0067] Traversing the scene operation information, extracting and mutually mapping the indicator feature values ​​based on the quality standard, and determining the homography matrix, where the homography matrix is ​​a mapping between at least two scene matrices;

[0068] Based on the homography matrix, an index interval evaluation is performed to determine the first evaluation result.

[0069] Ensure that the communication interface specifications between the intelligent assessment module and the software database are unified and bidirectional, including API calls, database connections, message queue subscriptions and other means; clarify the data structure, field meanings, encoding rules, etc. of the scenario operation information stored in the software database so that the intelligent assessment module can correctly parse and process it. Furthermore, configure necessary access rights to ensure that the intelligent assessment module can securely access the required data while preventing data leakage or malicious tampering;

[0070] According to the requirements, randomly select at least two different software operation scenarios, such as high concurrent access, network environment fluctuations, data exception processing, specific function usage, etc. Further, send a query request to the software database through the intelligent evaluation module, specify the evaluation scenario to be extracted and its corresponding operation time period, and the software database returns the corresponding scenario operation information according to the request, and performs data cleaning on the original scenario operation information obtained, removes duplicate or abnormal data, and performs necessary data conversion (such as timestamp conversion, unit standardization, etc.) to ensure the accuracy of subsequent analysis;

[0071] According to the pre-set quality standards (including rigid quality standards and flexible quality standards), relevant performance indicator characteristic values ​​are extracted from the scenario operation information. For example, for rigid quality standards, they may include system response time, resource utilization, error rate, etc.; for flexible quality standards, they may involve user experience, adaptability, maintainability, etc. Furthermore, for each scenario, the extracted indicator characteristic values ​​are organized into a scenario feature matrix, and a homography matrix is ​​constructed by calculating the similarity, correlation or transformation relationship between different scenario matrices. The homography matrix reflects the correspondence and change law of software performance indicators under different evaluation scenarios;

[0072] According to the eigenvalue distribution and change trend revealed by the homography matrix, each performance indicator is reasonably divided into intervals. The intervals can be determined based on statistical methods (such as quartiles, standard deviations, etc.). Furthermore, for each evaluation scenario, the indicator eigenvalues ​​in the scenario are mapped to the corresponding indicator intervals according to their mapping relationship in the homography matrix. By comparing the proportions or average values ​​of the same indicator falling into different intervals in different scenarios, the adaptability of the software in various scenarios is evaluated; then, the adaptability evaluation results of each scenario are integrated to form a first evaluation result, which includes the adaptability score of each scenario, the overall adaptability level, the scenario sensitivity analysis of the key performance indicators, etc.;

[0073] In addition, in actual operation, the random distribution of scenarios and feature states helps to simulate a wider range of operating conditions, thereby enhancing the generalization and adaptability of the evaluation results. In specific implementation, a variety of representative random evaluation scenarios that conform to the actual application environment are generated using methods such as probability models or Monte Carlo simulations. Furthermore, when dividing the index interval, the actual distribution characteristics of the indicator feature values ​​(such as normal distribution, skewed distribution, etc.) are considered. According to the distribution characteristics, a suitable interval division method is selected to ensure that the interval reflects the extreme conditions of the data while covering most of the observed values;

[0074] Based on the scene operation information in the software database, homography processing and adaptive evaluation technology are used to conduct a comprehensive and detailed analysis of the adaptability of the target software in random evaluation scenarios, and generate a first evaluation result with high fitness. In this way, when the scene is random and the distribution of the corresponding feature state is random, interval analysis is performed based on random distribution to improve the fitness of the result.

[0075] Furthermore, the performing of interval evaluation of indicators to determine the first evaluation result, the method of the present application includes:

[0076] Identifying the homography matrix, performing scene differentiation measurement based on the static evaluation branch, and determining a static evaluation result;

[0077] Identify the homography matrix, and based on the dynamic evaluation branch, perform dynamic trend analysis based on the scene feature interval and the indicator characteristic value interval to determine the dynamic evaluation result of the scene evolution;

[0078] The first evaluation result is determined based on the dynamic evaluation result and the static evaluation result.

[0079] A detailed interpretation of the constructed homography matrix is ​​conducted to analyze the mutual mapping relationship of various performance indicators in different evaluation scenarios and their transformation rules between different scenarios. The identification of important features in the homography matrix that reflect significant differences in software performance and affect adaptability, such as strong correlations between indicators, outliers in specific scenarios, and turning points of indicator changes;

[0080] Based on the distribution characteristics of each indicator in the homography matrix and the preset evaluation criteria, each performance indicator is divided into several intervals, such as excellent, good, average, poor, etc. For each evaluation scenario, the proportion, average value or median of the characteristic values ​​of each indicator in the scenario falling into the corresponding interval is calculated to form a differentiated feature vector of the scenario;

[0081] Comparative analysis of the differentiated feature vectors of each scene can be performed by clustering, distance measurement, ranking, etc. to quantify the degree of difference between scenes and obtain static evaluation results. Static evaluation results may include scene classification, scene difference score, key scene identification, etc. Furthermore, based on the feature relationship between scenes revealed by the homography matrix, scene intervals with similar features are divided. These intervals may be determined based on factors such as the similarity of performance indicators and the continuity of scene changes.

[0082] Perform trend analysis on the characteristic values ​​of each indicator within each scenario feature interval, including linear regression, time series analysis, autoregressive moving average model, etc., to reveal the dynamic trend of indicators as the scenario changes, and generate dynamic trend descriptions (such as growth, decline, fluctuation range, periodicity, etc.), key turning points, future forecasts and other information of indicators within each scenario feature interval to form dynamic evaluation results. Dynamic evaluation results may include scenario evolution paths, trend graphs of key indicators, prediction models, etc.

[0083] The static evaluation results and the dynamic evaluation results are organically integrated. The static evaluation results provide instant difference comparisons between scenarios, while the dynamic evaluation results reveal the evolution trend of scenarios over time or as conditions change. Combining the two, we can fully understand the adaptability of the software in different scenarios and its development and changes. Furthermore, based on the scenario difference score in the static evaluation results and the trend analysis in the dynamic evaluation results, an adaptability score is assigned to each scenario. The adaptability score can be a sub-item score of multiple sub-dimensions (such as stability, response speed, resource utilization efficiency, etc.); then, the adaptability scores of all scenarios are summarized, combined with the importance weights of the scenarios (such as business criticality, frequency of occurrence, etc.), and the overall adaptability score of the target software is calculated as the first evaluation result.

[0084] Through the above steps, based on the homography matrix, combined with static evaluation branches and dynamic evaluation branches, the adaptability of the target software in different scenarios is deeply analyzed to generate the first evaluation results that reflect the current status and foresee future trends, providing a scientific basis for software performance optimization, scenario adaptability improvement, risk warning, etc.

[0085] Furthermore, the method of determining the second evaluation result includes:

[0086] Read the operation control extreme value and risk resistance extreme value, where the extreme value interval is determined based on the upper and lower limits of the capacity;

[0087] Based on the software database, a predetermined number of risk operation records are randomly extracted, and the second evaluation result is evaluated and determined with the operation control extreme value and the risk confrontation extreme value as the baseline, wherein the elastic relaxation coefficient includes a coefficient multivariate group based on diversified elasticity indicators, and response efficiency and response accuracy are elasticity indicator evaluation elements.

[0088] According to the design specifications, industry standards, past experience or performance test results of the software system, clear upper and lower limits are set for the software's operation control capability and risk resistance capability, representing the target software's best performance under ideal conditions and its minimum tolerance under extreme pressure. Furthermore, for each capability indicator (such as response time, throughput, error rate, etc.), its extreme value interval is calculated based on the set upper and lower limits. The extreme value interval is the reasonable range of performance that the software should maintain during normal operation. Exceeding this range indicates the existence of performance bottlenecks or potential risks;

[0089] Clarify the definition criteria of risk operation records, including abnormal event triggering, performance indicators approaching or exceeding the extreme value interval, specific failure mode occurrence, etc., determine the number and method of randomly sampling risk operation records, ensure the representativeness of the sample, and consider factors such as time span, event type, and system load to ensure the diversity and balance of the sample. Furthermore, use the software database query interface to randomly extract a specified number of risk operation records from the database according to the preset screening conditions and sampling strategies. The records should include key information such as timestamp, event description, and relevant performance indicator data;

[0090] Check the actual values ​​of each performance indicator in each risk operation record one by one, compare it with the corresponding capacity extreme value interval, and determine whether the actual value exceeds the set upper and lower limits. Among them, for each record, calculate the two elastic indicators of response efficiency and response accuracy. Response efficiency involves the speed of processing requests, resource utilization, etc.; response accuracy is related to the correctness of the system processing requests, and the absence of false positives and negatives, etc.

[0091] Based on the calculated response efficiency and response accuracy, a comprehensive evaluation is performed using the coefficient multivariate of the multivariate elasticity index. The coefficient multivariate contains the weights and adjustment factors of each elasticity index, which is used to measure the relative importance of each index in the overall elasticity evaluation and the degree of its impact on system performance. The elastic relaxation coefficient of each record is obtained through weighted summation or other composite operations;

[0092] According to the elastic relaxation coefficient of each risk operation record, it is divided into different risk levels (such as low risk, medium risk, and high risk). The classification standard should match the business risk tolerance, compliance requirements, and emergency plans. Furthermore, the records of different risk levels are counted to analyze the frequency, duration, and impact range of risk events, and clarify the overall performance in responding to risks;

[0093] Then, taking into account the risk level distribution, risk event statistics, and the system's response efficiency and accuracy in the risk confrontation process, a second assessment result is formed, including risk profile summary, main risk factor analysis, elastic relaxation coefficient distribution, improvement suggestions, etc.; through the above steps, the target software's operation control capability and risk confrontation capability in the face of risk events are determined, that is, the second assessment result, which provides strong support for the software system's risk prevention and control, emergency response mechanism optimization, performance bottleneck investigation and other work.

[0094] Furthermore, if Figure 2 As shown, the method includes independent verification and mutual verification, and the method of the present application includes:

[0095] Interaction indicator threshold standard, performing attribution verification on the first evaluation result and the second evaluation result to determine an independent verification result;

[0096] Mapping and checking the first evaluation result and the second evaluation result to determine whether the deviation fluctuation range is satisfied, and determining a mutual verification result;

[0097] If the independent verification result is qualified and the mutual verification result is qualified, then the verification result is qualified.

[0098] The process including independent verification and mutual verification is intended to verify the rationality and consistency of the first evaluation result and the second evaluation result. According to industry standards, threshold standards are set for various interactive evaluation indicators. The interactive indicator threshold standards reflect the ideal range of various interactive performance indicators under normal operating conditions of the software. Further, all evaluation data related to interactive performance are extracted from the first evaluation result, including response time, interactive success rate, user experience score, etc.

[0099] Compare the interactive performance data in the first evaluation result with the set threshold standard. Further, if all data points fall within their respective threshold ranges, the evaluation result is considered to meet the interactive index threshold standard, the attribution verification is passed, and the independent verification result is recorded as qualified; otherwise, the index that does not meet the standard and its deviation value are recorded, and the independent verification result is unqualified;

[0100] Analyze the indicators associated with the first and second evaluation results, and clarify the logical relationship or causal effect between them. For example, response efficiency (the indicator in the first evaluation result) directly affects risk resistance (the indicator in the second evaluation result). In this way, a mapping relationship between the indicators is established, and a reasonable deviation fluctuation range is set for the mapped related indicators. The deviation fluctuation range should allow a certain degree of natural fluctuation or measurement error, and at the same time be able to identify significant inconsistencies or anomalies. The size of the interval can be determined based on the statistical characteristics of historical data, business sensitivity, and evaluation accuracy requirements;

[0101] Map the relevant indicator data in the first evaluation result to the indicator system corresponding to the second evaluation result, and compare the difference between the mapped data and the actual second evaluation result. Further, if the difference between all the mapped data points and the second evaluation result is within the set deviation fluctuation range, it is determined that the mutual verification result is qualified; otherwise, the indicator and its deviation value that exceed the fluctuation range are recorded, and the mutual verification result is unqualified;

[0102] Collect independent verification results and mutual verification results, record their respective pass / fail status and possible problem indicators. Further, if both the independent verification results and the mutual verification results are qualified, the comprehensive assessment verification result is qualified, indicating that there is no significant contradiction or abnormality between the first assessment result and the second assessment result, and the assessment process is credible. If any of the verification results is unqualified, the comprehensive assessment verification result is unqualified, and further investigation of the cause is required. It is necessary to redo part or all of the assessment work, or adjust the assessment method and parameters;

[0103] In the actual operation process, for the case of verification failure, in-depth analysis of the specific reasons leading to the verification failure, which may be the limitations of the evaluation method, deviations in data collection, unreasonable threshold setting, etc.; based on the analysis results, improvement measures are proposed, such as optimizing the evaluation model, correcting the data collection process, adjusting the threshold standard, etc. Furthermore, regardless of whether the verification results are qualified or not, a detailed verification report should be written to clearly explain the verification process, results and conclusions, provide improvement suggestions for unqualified situations, communicate with the project team, and provide support for subsequent decision-making; rigorous independent verification and mutual verification of the first evaluation results and the second evaluation results should be carried out to ensure the rationality and consistency of the evaluation data, and provide a reliable basis for the continuous improvement and optimization of the software system.

[0104] In summary, the beneficial effects of the embodiments of the present application are:

[0105] 1. By introducing diversified dimensions, adaptability of methods and randomness of scale, a comprehensive and multi-level evaluation of software operation is achieved, covering rigid and flexible quality standards, single and interactive evaluation dimensions, ensuring the completeness and accuracy of the evaluation.

[0106] 2. Compare the assessment scenarios and conduct intelligent analysis in combination with the software database to ensure that the assessment is close to the actual operating environment, make full use of big data resources, and improve the pertinence and authenticity of the assessment.

[0107] 3. Through the acquisition of homography matrix and index interval evaluation, combined with static and dynamic evaluation branches, the software's operational adaptive adjustment capability and risk resistance capability are specifically evaluated. The introduction of concepts such as elastic relaxation coefficient strengthens the evaluation of the software's ability to cope with complex environmental changes and risk events, realizes dynamic tracking and trend analysis of the software's operating status, and improves the timeliness and foresight of the evaluation.

[0108] 4. Due to the adoption of interactive indicator threshold standards, the first evaluation result and the second evaluation result are subjected to attribution verification to determine the independent verification result; the first evaluation result and the second evaluation result are mapped and checked to determine whether they meet the deviation fluctuation range and determine the mutual verification result; if the independent verification result is qualified and the mutual verification result is qualified, then the verification result is qualified. The first evaluation result and the second evaluation result are rigorously independently verified and mutually verified to ensure the rationality and consistency of the evaluation data, providing a reliable basis for the continuous improvement and optimization of the software system.

[0109] Embodiment 2

[0110] Based on the same inventive concept as the software operation information evaluation method based on artificial intelligence in the above-mentioned embodiment, Figure 3 As shown, the embodiment of the present application provides a software operation information evaluation system based on artificial intelligence, wherein the system includes:

[0111] The standard determination module M100 is used to determine the quality standard of the target software, wherein the quality standard includes a rigid quality standard and a flexible quality standard, and the quality measurement dimension includes a single evaluation dimension and an interactive evaluation dimension;

[0112] An operation data determination module M200 is used to determine an evaluation scenario based on the target software and determine effective scenario operation data, wherein the evaluation scenario is at least two random ones;

[0113] The evaluation and analysis module M300 is used to connect to the software database, and based on the quality standard, perform evaluation and analysis based on the evaluation scenario in the intelligent evaluation module, including homography matrix acquisition and index interval evaluation based on the scenario interval, to determine a first evaluation result;

[0114] A capability evaluation module M400 is used to evaluate the operation adaptive adjustment capability and risk resistance capability of the target software in combination with the software database, and determine a second evaluation result, wherein the second evaluation result is marked with an elastic relaxation coefficient;

[0115] A verification module M500, configured to verify the first evaluation result and the second evaluation result, including independent verification and mutual verification, to determine a verification result;

[0116] The mean calculation module M600 is used to calculate the mean of the first evaluation result and the second evaluation result as the target evaluation result if the verification result is qualified.

[0117] Furthermore, the mean value calculation module M600 is also used to execute the following method:

[0118] Conducting homology search on the target software to determine a set of similar software;

[0119] Execute inter-class function ranking and assembly calculation of the target software and the class software set to determine the class evaluation result, wherein the assembly calculation principle is a weighted calculation based on the priority of functions;

[0120] A software evaluation result is determined based on the target evaluation result and the class evaluation result.

[0121] Furthermore, the standard determination module M100 is used to execute the following method:

[0122] Determine collaborative software of the target software and determine collaborative features, wherein the collaborative features are determined based on software interactive functions;

[0123] A first weight distribution is performed based on interaction dominance, and a second weight distribution is performed based on interaction relevance, wherein the relative dominance based on software interaction is marked;

[0124] Determine a multivariate evaluation index of the interactive evaluation dimension, and perform hierarchical weighting processing based on the first weight distribution and the second weight distribution.

[0125] Furthermore, the intelligent evaluation module is communicatively connected with the software database, and includes a homography processing unit and an adaptive evaluation unit, wherein the adaptive evaluation unit includes a static evaluation branch and a dynamic evaluation branch, and an evaluation analysis based on the evaluation scenario is performed in the intelligent evaluation module. The software operation information evaluation system based on artificial intelligence is also used to perform the following method:

[0126] Based on the software database, extracting scenario operation information based on the evaluation scenario;

[0127] Traversing the scene operation information, extracting and mutually mapping the indicator feature values ​​based on the quality standard, and determining the homography matrix, where the homography matrix is ​​a mapping between at least two scene matrices;

[0128] Based on the homography matrix, an index interval evaluation is performed to determine the first evaluation result.

[0129] Furthermore, the index interval evaluation is performed to determine the first evaluation result, and the software operation information evaluation system based on artificial intelligence is also used to execute the following method:

[0130] Identifying the homography matrix, performing scene differentiation measurement based on the static evaluation branch, and determining a static evaluation result;

[0131] Identify the homography matrix, and based on the dynamic evaluation branch, perform dynamic trend analysis based on the scene feature interval and the indicator characteristic value interval to determine the dynamic evaluation result of the scene evolution;

[0132] The first evaluation result is determined based on the dynamic evaluation result and the static evaluation result.

[0133] Furthermore, the capability assessment module M400 is used to perform the following method:

[0134] Read the operation control extreme value and risk resistance extreme value, where the extreme value interval is determined based on the upper and lower limits of the capacity;

[0135] Based on the software database, a predetermined number of risk operation records are randomly extracted, and the second evaluation result is evaluated and determined with the operation control extreme value and the risk confrontation extreme value as the baseline, wherein the elastic relaxation coefficient includes a coefficient multivariate group based on diversified elasticity indicators, and response efficiency and response accuracy are elasticity indicator evaluation elements.

[0136] Furthermore, the verification module M500 is used to perform the following method:

[0137] Interaction indicator threshold standard, performing attribution verification on the first evaluation result and the second evaluation result to determine an independent verification result;

[0138] Mapping and checking the first evaluation result and the second evaluation result to determine whether the deviation fluctuation range is satisfied, and determining a mutual verification result;

[0139] If the independent verification result is qualified and the mutual verification result is qualified, then the verification result is qualified.

[0140] In summary, any step can be stored as a computer instruction or program in an unlimited computer memory, and can be called and recognized by an unlimited computer processor, and no unnecessary restrictions are made here.

[0141] Furthermore, the above-mentioned technical scheme only reflects the preferred technical scheme of the technical scheme of the embodiment of the present application. Some changes that may be made to some parts thereof by technicians in this technical field all reflect the novel principles of the embodiment of the present application. Obviously, technicians in this field can make various changes and modifications to the present application without departing from the scope of the present application.

Claims

1. A software operation information evaluation method based on artificial intelligence, characterized in that: The method comprises: Determine the quality standard of the target software, wherein the quality standard includes a rigid quality standard and a flexible quality standard, and the quality measurement dimension includes a single evaluation dimension and an interactive evaluation dimension; Determine an evaluation scenario based on the target software, and determine valid scenario operation data, wherein the evaluation scenario is at least two random ones; Connecting to a software database, taking the quality standard as a benchmark, performing an evaluation analysis based on the evaluation scenario in an intelligent evaluation module, including obtaining a homography matrix and evaluating the indicators based on the scenario interval, and determining a first evaluation result; In combination with the software database, the target software is evaluated for its operation adaptive adjustment capability and risk resistance capability to determine a second evaluation result, wherein the second evaluation result is marked with an elastic relaxation coefficient; Verifying the first evaluation result and the second evaluation result, including independent verification and mutual verification, to determine a verification result; If the verification result is qualified, the first evaluation result and the second evaluation result are averaged and used as the target evaluation result.

2. The method according to claim 1, characterized in that After obtaining the target assessment results, including: Conducting homology search on the target software to determine a set of similar software; Execute inter-class function ranking and assembly calculation of the target software and the class software set to determine the class evaluation result, wherein the assembly calculation principle is a weighted calculation based on the priority of functions; A software evaluation result is determined based on the target evaluation result and the class evaluation result.

3. The method according to claim 1, characterized in that The quality measurement dimensions include interactive evaluation dimensions, including: Determining collaborative software of the target software and determining collaborative features, wherein the collaborative features are determined based on software interactive functions; A first weight distribution is performed based on interaction dominance, and a second weight distribution is performed based on interaction relevance, wherein the relative dominance based on software interaction is marked; Determine a multivariate evaluation index of the interactive evaluation dimension, and perform hierarchical weighting processing based on the first weight distribution and the second weight distribution.

4. The method according to claim 1, characterized in that The intelligent evaluation module is in communication with the software database, and includes a homography processing unit and an adaptive evaluation unit. The adaptive evaluation unit includes a static evaluation branch and a dynamic evaluation branch. The evaluation analysis based on the evaluation scenario is performed in the intelligent evaluation module, including: Based on the software database, extracting scenario operation information based on the evaluation scenario; Traversing the scene operation information, extracting and mutually mapping the indicator feature values ​​based on the quality standard, and determining the homography matrix, where the homography matrix is ​​a mapping between at least two scene matrices; Based on the homography matrix, an index interval evaluation is performed to determine the first evaluation result.

5. The method according to claim 4, characterized in that The performing interval evaluation of indicators to determine the first evaluation result includes: Identifying the homography matrix, performing scene differentiation measurement based on the static evaluation branch, and determining a static evaluation result; Identify the homography matrix, and based on the dynamic evaluation branch, perform dynamic trend analysis based on the scene feature interval and the indicator characteristic value interval to determine the dynamic evaluation result of the scene evolution; The first evaluation result is determined based on the dynamic evaluation result and the static evaluation result.

6. The method according to claim 1, characterized in that The determining of the second evaluation result comprises: Read the operation control extreme value and risk resistance extreme value, where the extreme value interval is determined based on the upper and lower limits of the capacity; Based on the software database, a predetermined number of risk operation records are randomly extracted, and the second evaluation result is evaluated and determined with the operation control extreme value and the risk confrontation extreme value as the baseline, wherein the elastic relaxation coefficient includes a coefficient multivariate group based on diversified elasticity indicators, and response efficiency and response accuracy are elasticity indicator evaluation elements.

7. The method according to claim 1, characterized in that The independent verification and mutual verification include: Interaction indicator threshold standard, performing attribution verification on the first evaluation result and the second evaluation result to determine an independent verification result; Mapping and checking the first evaluation result and the second evaluation result to determine whether the deviation fluctuation range is satisfied, and determining a mutual verification result; If the independent verification result is qualified and the mutual verification result is qualified, then the verification result is qualified.

8. A software operation information evaluation system based on artificial intelligence, characterized in that: A method for evaluating software operation informatization based on artificial intelligence according to any one of claims 1 to 7, the system comprising: A standard determination module, used to determine the quality standard of the target software, wherein the quality standard includes a rigid quality standard and a flexible quality standard, and the quality measurement dimension includes a single evaluation dimension and an interactive evaluation dimension; An operation data determination module, used to determine an evaluation scenario based on the target software and determine effective scenario operation data, wherein the evaluation scenario is at least two random ones; An evaluation and analysis module, which is used to connect to a software database, and to perform evaluation and analysis based on the evaluation scenario in an intelligent evaluation module based on the quality standard, including homography matrix acquisition and index interval evaluation based on the scenario interval, to determine a first evaluation result; A capability assessment module, used to evaluate the operation adaptive adjustment capability and risk resistance capability of the target software in combination with the software database, and determine a second assessment result, wherein the second assessment result is marked with an elastic relaxation coefficient; A verification module, used to verify the first evaluation result and the second evaluation result, including independent verification and mutual verification, to determine a verification result; The mean calculation module is used to calculate the mean of the first evaluation result and the second evaluation result as the target evaluation result if the verification result is qualified.

Images