System executive force evaluation method, device and equipment, storage medium and product

Through the automated system execution evaluation method, keywords and weights are extracted and scored in association data are calculated, the problem of lack of real-time monitoring of system execution evaluation in the existing technology is solved, and the automation of system execution evaluation is realized, which reduces costs and is timely. Discover risks.

CN119047903APending Publication Date: 2024-11-29CHINA CITIC BANK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411033877.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The lack of real-time monitoring and dynamic updates of the system execution ability evaluation in the prior art, resulting in the failure to expose risks in a timely manner, which is costly and time-consuming.

Method used

By obtaining the content of the target system and execution data, extracting keywords, determining the execution control point list and its weight, and correlating the data to calculate the execution score, and achieving automated evaluation.

Benefits of technology

Reduce the cost of manual compliance management, promptly discover and solve problems in system implementation, enhance system execution, and reduce risk exposure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119047903A_ABST
    Figure CN119047903A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to the technical field of artificial intelligence, and provides a system execution force evaluation method and device, equipment, a storage medium and a product, and the method comprises the steps: obtaining target system content and target system execution data; performing keyword extraction on the content of the target system to obtain an execution control point list of the target system, and determining a weight and a monitoring rule of each execution control point in the execution control point list; associating the target system execution data with execution control points in the execution control point list; determining a quantitative score of the execution control point associated with the target system execution data according to the monitoring rule of the execution control point; performing calculation according to the quantitative score of the target system execution data association execution control point and the weight of the target system execution data association execution control point to obtain an execution score of the target system; and determining an executive force evaluation result of the target system according to the execution score of the target system. Through the embodiment of the invention, the manual compliance management cost can be reduced, the compliance risk can be avoided in advance, and the system execution force can be enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and in particular to a method, device, equipment, storage medium and product for evaluating system execution. Background Art

[0002] System execution evaluation refers to an objective and comprehensive assessment and analysis of the implementation of internal rules and regulations of an organization, the effectiveness of implementation, and the problems and deficiencies in the implementation process. In recent years, faced with the risk of continued high pressure and normalization of supervision, the system, as the cornerstone and standard of corporate business management, its execution restricts the industry's stable, sustainable and high-quality development. At present, the evaluation of system implementation is mainly carried out through manual review and post-event supervision. These methods are time-consuming and labor-intensive, with high human and material costs, and the implementation status is not monitored in real time and dynamically updated, resulting in the problem of risks not being exposed in a timely manner. Summary of the invention

[0003] In response to the above-mentioned problems in the prior art, the purpose of the embodiments of this specification is to provide a method, device, equipment, storage medium and product for evaluating the execution of a system, so as to solve the problem in the prior art that the execution status is not monitored in real time and updated dynamically, resulting in the risk not being exposed in time.

[0004] In order to solve the above technical problems, the specific technical solutions of the embodiments of this specification are as follows:

[0005] On the one hand, the embodiments of this specification provide a method for evaluating the execution of a system, the method comprising:

[0006] Obtain the content of the target system and the data on the implementation of the target system;

[0007] Extract keywords from the content of the target system, obtain a list of execution control points of the target system, and determine the weight and monitoring rules of each execution control point in the list of execution control points, wherein the list of execution control points includes conduction learning control points, risk control points, and violation problem control points;

[0008] Associating the target system execution data with the execution control points in the execution control point list;

[0009] Determining the quantitative score of the execution control point associated with the target system execution data according to the monitoring rules of the execution control point;

[0010] Obtaining an execution score of the target system according to the quantitative scores of the execution control points associated with the target system execution data and the weights of the execution control points associated with the target system execution data;

[0011] The execution evaluation result of the target system is determined according to the execution score of the target system.

[0012] Furthermore, the control points of the conduction learning category include training conditions, system reading volume and material download volume;

[0013] The risk-type control points include control points of multiple risk levels;

[0014] The violation control points include internal and external inspection issues, production incidents and negative suggestions on system implementation.

[0015] Furthermore, the keyword extraction of the target system content is performed to obtain the execution control point list of the target system, including:

[0016] Extracting a first keyword set of the target system from the target system content using a predefined regular expression;

[0017] Matching the target system content with a preset associated word list to obtain a second keyword set for the target system, wherein the associated word list is set based on the target system content;

[0018] Merging the first keyword set and the second keyword set to obtain a third keyword set;

[0019] De-duplication processing is performed on the third keyword set to obtain an execution control point list of the target system;

[0020] The execution control point list is input into a pre-trained classification model to obtain the category of each execution control point.

[0021] Furthermore, determining the weight of each execution control point in the execution control point list includes:

[0022] Constructing a judgment matrix according to the relative importance of each execution control point, wherein the judgment matrix includes a single-layer judgment matrix and a multi-layer judgment matrix;

[0023] Calculate the first maximum eigenvalue and the corresponding first eigenvector of the single-layer judgment matrix,

[0024] Performing a consistency check on the first eigenvector according to the first maximum eigenvalue, and if the check passes, using the first eigenvector as a first weight vector;

[0025] Calculate the second largest eigenvalue and the corresponding second eigenvector of the multi-layer judgment matrix;

[0026] Performing a consistency check on the second eigenvector according to the second maximum eigenvalue, and if the check passes, using the second eigenvector as a second weight vector;

[0027] The weight of each execution control point is calculated according to the first weight vector and the second weight vector.

[0028] Furthermore, before extracting keywords from the target system content to obtain the execution control point list of the target system, the process further includes:

[0029] Split the target system content to obtain standardized texts of chapters, sections, and articles;

[0030] Keyword recognition is performed on the standardized text, and system entries containing execution control points are screened according to the recognition results, while system entries not containing execution control points are eliminated.

[0031] Furthermore, associating the target system execution data with the execution control points in the execution control point list includes:

[0032] Structuring the target system execution data to obtain structured execution data;

[0033] Performing text analysis on the structured execution data using a text analysis model to obtain a text analysis result;

[0034] Extract keywords from the text analysis results according to the execution control point list to obtain a keyword list of the execution data;

[0035] Calculating the matching degree between each keyword in the keyword list and each execution control point in the execution control point list;

[0036] The execution control point with the highest matching degree for each keyword is used as the associated execution control point of the keyword.

[0037] On the other hand, an embodiment of the present specification provides a system execution evaluation device, the device comprising:

[0038] An acquisition module is used to acquire the target system content and target system execution data;

[0039] An execution control point extraction and weight determination module is used to extract keywords from the target system content, obtain the execution control point list of the target system and determine the weight and monitoring rules of each execution control point in the execution control point list, wherein the execution control point list includes conduction learning control points, risk control points and violation problem control points;

[0040] An association module, used for associating the target system execution data with the execution control points in the execution control point list;

[0041] An execution control point score quantification module, used to determine the quantitative score of the execution control point associated with the target system execution data according to the monitoring rules of the execution control point;

[0042] An execution score calculation module, used for calculating the execution score of the target system according to the quantitative scores of the execution control points associated with the target system execution data and the weights of the execution control points associated with the target system execution data;

[0043] The execution capability evaluation module is used to determine the execution capability evaluation result of the target system according to the execution score of the target system.

[0044] On the other hand, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the instructions of any one of the above methods are executed.

[0045] On the other hand, an embodiment of the present specification further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of a computer device, the computer program executes instructions of any one of the above-described methods.

[0046] On the other hand, the embodiments of this specification further provide a computer program product, which, when executed by a processor of a computer device, executes instructions of any one of the above methods.

[0047] One or more technical solutions provided by some embodiments of this specification have at least the following technical effects:

[0048] This manual first extracts keywords from the content of the target system, obtains the list of execution control points of the target system and determines the weight and monitoring rules of each execution control point in the list of execution control points, then associates the execution data of the target system with the execution control points in the list of execution control points, and then determines the quantitative score of the execution control points associated with the execution data of the target system according to the monitoring rules of the execution control points, and finally calculates the execution score of the target system according to the quantitative score of the execution control points associated with the execution data of the target system and the weight of the execution control points associated with the execution data of the target system, thereby determining the execution evaluation result of the target system according to the execution score of the target system. Therefore, through the above scheme, the execution of the system can be automatically evaluated according to the execution data of the system, thereby reducing the cost of manual compliance management, and enhancing the execution of the system according to the evaluation results, and then timely discovering and solving problems in the implementation of the system before risks are exposed.

[0049] The above description is only an overview of the technical solutions of some embodiments of this specification. In order to more clearly understand the technical means of some embodiments of this specification, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of this specification more obvious and easy to understand, the following specifically cites the preferred embodiments and describes them in detail in conjunction with the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic diagram showing a flow chart of a method for evaluating system execution in some embodiments of this specification;

[0052] Figure 2 A schematic diagram of a process of extracting keywords from the target system content to obtain an execution control point list of the target system in some embodiments of this specification is shown;

[0053] Figure 3 A schematic diagram of a process of determining the weight of each execution control point in the execution control point list in some embodiments of the present specification is shown;

[0054] Figure 4 A schematic diagram of a process for preprocessing target system content in some embodiments of this specification is shown;

[0055] Figure 5 A schematic diagram of a process of associating the target system execution data with the execution control points in the execution control point list in some embodiments of this specification is shown;

[0056] Figure 6 A schematic diagram showing the structure of a system execution evaluation device in some embodiments of this specification is shown;

[0057] Figure 7 A schematic diagram of the structure of a computer device in this specification is shown.

[0058] Description of the accompanying symbols:

[0059] 601. Get module;

[0060] 602. Execute a control point extraction and weight determination module;

[0061] 603, association module;

[0062] 604. Execute a control point score quantification module;

[0063] 605. Execute a score calculation module;

[0064] 606. Execution evaluation module;

[0065] 702. Computer equipment;

[0066] 704, processor;

[0067] 706. Memory;

[0068] 708, driving mechanism;

[0069] 710, input / output module;

[0070] 712. Input devices;

[0071] 714. Output devices;

[0072] 716. Presentation equipment;

[0073] 718. Graphical user interface;

[0074] 720, network interface;

[0075] 722. Communication link;

[0076] 724. Communication bus. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0078] In order to solve the above problems, the embodiments of this specification provide a method for evaluating the execution of a system. Figure 1 It is a flowchart of a method for evaluating the execution of a system provided in an embodiment of this specification. This specification provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiment or the accompanying drawings or in parallel.

[0079] It should be noted that the terms "first", "second", etc. in this specification and claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0080] Reference Figure 1 As shown, the embodiment of this specification provides a method for evaluating the execution of a system, and the method includes:

[0081] S101: Obtain the target system content and target system implementation data;

[0082] S102: extract keywords from the target system content, obtain an execution control point list of the target system, and determine the weight and monitoring rules of each execution control point in the execution control point list, wherein the execution control point list includes conduction learning control points, risk control points, and violation problem control points;

[0083] S103: Associating the target system execution data with the execution control points in the execution control point list;

[0084] S104: Determine the quantitative score of the execution control point associated with the target system execution data according to the monitoring rule of the execution control point;

[0085] S105: Obtaining the execution score of the target system according to the quantitative scores of the execution control points associated with the target system execution data and the weights of the execution control points associated with the target system execution data;

[0086] S106: Determine the execution evaluation result of the target system according to the execution score of the target system.

[0087] This manual first extracts keywords from the content of the target system, obtains the list of execution control points of the target system and determines the weight and monitoring rules of each execution control point in the list of execution control points, then associates the execution data of the target system with the execution control points in the list of execution control points, and then determines the quantitative score of the execution control points associated with the execution data of the target system according to the monitoring rules of the execution control points, and finally calculates the execution score of the target system according to the quantitative score of the execution control points associated with the execution data of the target system and the weight of the execution control points associated with the execution data of the target system, thereby determining the execution evaluation result of the target system according to the execution score of the target system. Therefore, through the above scheme, the execution of the system can be automatically evaluated according to the execution data of the system, thereby reducing the cost of manual compliance management, and enhancing the execution of the system according to the execution evaluation results, thereby timely discovering and solving problems existing in the implementation of the system before risks are exposed.

[0088] In some embodiments, the target system content can be obtained from the system database, wherein the target system content includes the system name, system text, release time, validity and version number, etc. The target system execution data is the data involved in the system release, execution, and revision process, which can be obtained from the system middle platform, compliance rectification system, IT service management system or OA system. Since the data formats of different systems are different, it is necessary to perform ETL processing on the execution data from different source systems to obtain execution data in a unified format to provide data support for the calculation of subsequent execution scores. The formatted execution data is stored in the system middle platform system database, and a data basis is provided for intelligent indexing by constructing indexes, tags, features, etc. Semi-structured and unstructured data in the execution data, such as emergency drill reports, are structured and converted into structured data.

[0089] System execution control points refer to the key control links or nodes set up in the internal control system of an organization to ensure that the system is effectively implemented. These control points usually involve supervision, inspection and evaluation of the implementation of the system to ensure that the processes, standards and requirements stipulated in the system are strictly followed. The setting of system execution control points can help system managers to timely discover and correct deviations and errors in the process of system implementation, and improve the efficiency and effectiveness of system implementation. At the same time, execution control points can also provide system managers with feedback and information on the implementation of the system, helping system managers to continuously improve and optimize system design. In some embodiments of the present specification, execution control points include conduction and learning control points, risk control points and violation problem control points, wherein the conduction and learning control points refer to indicators that measure whether the system manager promotes and guides the implementers of the system to conduct training and learning, or the implementers' independent learning after the system is released, including training status, system reading volume and material download volume; the risk control points refer to key risk control points in the implementation process of the system or matters that are of high regulatory concern, including high-risk control points and low-risk control points, wherein high-risk control points have a higher priority than low-risk control points, such as mandatory requirements, which will have major risks or impacts if not promptly controlled, and low-risk control points such as reminder-type matters, which will not have major short-term risks if not promptly controlled, or matters that can be controlled by human control in the short term; the violation problem control points include internal and external inspection issues, production incidents and negative suggestions for system implementation, and violation problem control points indicate that the system is not implemented properly.

[0090] In some embodiments, reference Figure 2 , extracting keywords from the target system content to obtain a list of execution control points for the target system, including:

[0091] S201: extracting a first keyword set of the target system from the target system content using a predefined regular expression;

[0092] S202: Matching the target system content with a preset associated word list to obtain a second keyword set for the target system, wherein the associated word list is set based on the target system content;

[0093] S203: merging the first keyword set and the second keyword set to obtain a third keyword set;

[0094] S204: performing deduplication processing on the third keyword set to obtain an execution control point list of the target system;

[0095] S205: Input the execution control point list into a pre-trained classification model to obtain a category of each execution control point.

[0096] It can be understood that, according to the existing empirical rules for distinguishing execution control points, the empirical rules are abstracted into regular expressions, and the key phrases / words are extracted from the content of the target system using regular expressions, and the extracted key phrases / words are used as the first keyword set of the target system. In order to accurately determine the comprehensive execution control points, a list of associated words can also be set according to the content of the target system, and the fields / words matching the terms in the associated word list are searched through the content of the target system as the execution control points. Thus, by setting regular expressions and associated word lists, the extracted execution control points can be made more comprehensive and avoid omissions, thereby providing reliable data support for the subsequent execution score calculation and ensuring the accuracy of the execution score. Since regular matching and associated word matching may generate duplicate control points, after merging the first keyword set and the second keyword set to obtain the third keyword set, it is necessary to perform deduplication processing on the third keyword set to remove duplicate execution control points, and finally obtain the execution control point list of the target system. Since the importance of each execution control point is different, in order to facilitate the subsequent determination of the weight, it is necessary to determine the category of each execution control point in the execution control point list. In some embodiments, the execution control point list may be classified using a pre-trained classification model, wherein the classifier may include any model capable of classification, such as logistic regression, naive Bayes, support vector machine, or deep learning model, which is not limited in this specification.

[0097] In some embodiments, reference Figure 3 , the determining the weight of each execution control point in the execution control point list includes:

[0098] S301: constructing a judgment matrix according to the relative importance of each execution control point, wherein the judgment matrix includes a single-layer judgment matrix and a multi-layer judgment matrix.

[0099] According to the content of the execution control point list, the execution control point list can be regarded as a multi-level model structure, that is, the first level is the execution control point category, in which the elements include conduction learning control point U1, risk control point U2 and violation problem control point U3, and the second level is the specific control points under each category, in which the elements include training situation U11, system reading volume U12, data download volume U13, high risk control point U21, and low risk control point U22. The judgment matrix is ​​used to represent the relative importance of the elements of each layer, and the relative importance can be determined based on expert consensus, literature knowledge, etc. For any two elements in the same level, the numbers 1-9 and their reciprocals can be used as scales for pairwise comparison to indicate their relative importance. For example, if the risk control point U2 is very important relative to the conduction learning control point U1, then it can be assigned a value of 7, and the importance of the conduction learning control point U1 relative to the risk control point U2 is 1 / 7. Thus, the judgment matrix of the first level can be constructed, and its matrix size is 3×3. Similarly, a judgment matrix of the second layer can be constructed, which represents the relative importance of the second layer to the first layer. The judgment matrix of the second layer is a multi-layer judgment matrix, the number of which is 3, and the size of each matrix is ​​5×5.

[0100] S302: Calculate the first maximum eigenvalue and the corresponding first eigenvector of the single-layer judgment matrix.

[0101] S303: Perform a consistency check on the first eigenvector according to the first maximum eigenvalue, and if the check passes, use the first eigenvector as a first weight vector.

[0102] After constructing the judgment matrix, a consistency check is required to ensure the rationality of the judgment matrix. Specifically, first calculate the consistency index value CI based on the first maximum eigenvalue, then find the corresponding average random consistency index value RI from the known average random consistency index table based on the consistency index value CI, and finally calculate the consistency ratio CR based on CI and RI. When CR is less than 0.1, it indicates that the judgment matrix is ​​consistent, otherwise it is necessary to adjust the element values ​​of the judgment matrix and re-perform the consistency check until CR is less than 0.1. The consistency index value CI and the consistency ratio CR are calculated by the following formulas:

[0103]

[0104] Among them, CI represents the consistency index value, λ represents the first maximum eigenvalue, n represents the number of elements in the first layer, RI represents the average random consistency index value, and CR represents the consistency ratio.

[0105] S304: Calculate the second maximum eigenvalue and the corresponding second eigenvector of the multi-layer judgment matrix.

[0106] S305: Perform a consistency check on the second eigenvector according to the second maximum eigenvalue, and if the check passes, use the second eigenvector as a second weight vector.

[0107] S306: Calculate the weight of each execution control point according to the first weight vector and the second weight vector.

[0108] The weight of each execution control point can be obtained by multiplying and adding the first weight vector and the second weight vector. An example of weight distribution of each execution control point is shown in Table 1.

[0109] Table 1

[0110]

[0111] It should be noted that violation control points are deducted based on the total score of the first two control points, and no weight is involved.

[0112] In some embodiments, reference Figure 4 , before extracting keywords from the target system content to obtain the execution control point list of the target system, it also includes:

[0113] S401: splitting the target system content to obtain standardized texts of chapters, sections, and articles;

[0114] S402: Perform keyword recognition on the standardized text, filter out the system entries containing execution control points according to the recognition results, and remove the system entries that do not contain execution control points.

[0115] It can be understood that, based on the content of the system, quantifiable system items with execution control points are screened out. These items are execution items that can be refined into rules and standardized data, which are used for subsequent analysis and monitoring of their implementation effects. Items that are irrelevant to system implementation and cannot be quantified are eliminated, such as universal clauses, general provisions, annexes, some term definitions, principles or frameworks, and other high-level requirements in the system, which will not be subject to execution monitoring.

[0116] In some embodiments, reference Figure 5 , the associating the target system execution data with the execution control points in the execution control point list includes:

[0117] S501: Structuring the target system execution data to obtain structured execution data;

[0118] S502: Performing text analysis on the structured execution data using a text analysis model to obtain a text analysis result;

[0119] S503: extracting keywords from the text analysis result according to the execution control point list to obtain a keyword list of the execution data;

[0120] S504: Calculate the matching degree between each keyword in the keyword list and each execution control point in the execution control point list;

[0121] S505: Taking the execution control point with the highest matching degree with each keyword as the associated execution control point of the keyword.

[0122] It can be understood that for the target system execution data, it is necessary to determine the corresponding execution control point category, so that the execution score of the execution data can be determined according to the execution control point weight, so it is necessary to associate the execution data with the execution control points in the execution control point list. In some embodiments, the text analysis model can be any model that can perform text analysis, such as the Bert model, and this specification does not limit this. By extracting keywords from the text analysis results according to the execution control point list, the execution control points contained in the execution data can be quickly and accurately determined, thereby improving the accuracy of the execution score. Specifically, the text analysis results can be traversed, the text analysis results can be split according to punctuation marks, and the split fields or statements can be screened according to the contents of the execution control point list to obtain a keyword list for the execution data, and the matching degree between the keywords in the keyword list and the execution control points in the execution control point list is calculated, and the execution control point with the highest matching degree for each keyword is used as the associated execution control point of the keyword.

[0123] In some embodiments, the monitoring rules of the execution control point can be set through expert consensus, system monitoring guidelines, literature knowledge, etc., avoiding subjective judgment based on personal experience and knowledge in the prior art, so as to ensure the scientificity and reliability of the source of the monitoring rules. For the conduction learning control point, its monitoring rules can be: judging whether the person in charge of the system uploads training materials, whether the execution department reads the relevant system, whether the person in charge of the execution downloads the relevant materials according to the execution data, judging whether the system reading volume reaches the preset standard in the monitoring rules according to the total number of the execution department and the system reading volume, such as the system reading volume reaches 60%, judging whether the data download volume reaches the preset standard in the monitoring rules according to the total number of the execution responsible person and the data download volume, such as the data download volume reaches 60%, if the elements in the conduction learning control point all meet the standards in the monitoring rules, then the score of this type of control point can be calculated according to the weight. If the training situation meets the monitoring standard, its score is 0.2*0.8*100=16, if the training situation does not meet the monitoring standard, it means that the person in charge of the system has not uploaded the training materials, and its score is 0.

[0124] Examples of quantitative scores of execution control points in some embodiments of this specification are shown in Table 2. After determining the quantitative scores of the execution control points associated with the execution data of the target system according to the monitoring rules of the execution control points, the execution scores of the corresponding execution control points can be obtained by combining the weights of the corresponding execution control points, and the execution scores of the execution control points associated with the target system can be added to obtain the execution score of the target system. It should be noted that in some embodiments, the total score of the conduction learning control points and the risk control points is 100 points. If there are violations in the target system, the execution score is deducted according to the violations and the corresponding deduction standards. Finally, the execution power of the target system is evaluated according to the execution score of the target system. The execution power can be evaluated by setting a threshold. If the execution score exceeds the threshold, it indicates that the execution power of the target system meets the standard, otherwise it does not meet the standard. Optionally, an execution score report of the target system can be generated based on the execution data and the execution score. The report content may include statistics on the conduction learning of the system, the execution score of the system, and details of the violations involved in the system. Through the execution score report, the weaknesses of the system in terms of execution can be effectively understood, and the execution of which areas has more problems and which departments have a lower degree of execution can be analyzed. In addition, it is also possible to link the rectification data of compliance issues, connect the system with internal and external inspections, form a post-supervision mechanism and a digital monitoring system management model, and provide feedback and adjustment basis for the implementation of various systems.

[0125] Table 2

[0126]

[0127] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the acquisition, storage, use, and processing of data in the technical solutions described in the embodiments of this application are in compliance with relevant regulations.

[0128] Based on the above-mentioned method for evaluating the execution of a system, an embodiment of this specification also provides a corresponding device for evaluating the execution of a system. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiment of this specification and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiment of this specification is as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0129] Specifically, Figure 6 This is a schematic diagram of the module structure of an embodiment of a system execution evaluation device provided in the embodiment of this specification, referring to Figure 6 As shown, a system execution evaluation device provided in an embodiment of this specification includes:

[0130] An acquisition module 601 is used to acquire the target system content and target system execution data;

[0131] An execution control point extraction and weight determination module 602 is used to extract keywords from the target system content, obtain an execution control point list of the target system, and determine the weight and monitoring rules of each execution control point in the execution control point list, wherein the execution control point list includes conduction learning control points, risk control points, and violation problem control points;

[0132] An association module 603 is used to associate the target system execution data with the execution control points in the execution control point list;

[0133] An execution control point score quantification module 604 is used to determine the quantitative score of the execution control point associated with the target system execution data according to the monitoring rules of the execution control point;

[0134] An execution score calculation module 605 is used to calculate the execution score of the target system according to the quantitative scores of the execution control points associated with the target system execution data and the weights of the execution control points associated with the target system execution data;

[0135] The execution evaluation module 606 is used to determine the execution evaluation result of the target system according to the execution score of the target system.

[0136] The beneficial effects obtained by the device provided in the embodiments of this specification are consistent with the beneficial effects obtained by the above method and will not be repeated here.

[0137] Reference Figure 7 As shown, based on the above-mentioned method for evaluating the execution of a system, a computer device 702 is also provided in an embodiment of this specification, wherein the above-mentioned method runs on the computer device 702. The computer device 702 may include one or more processors 704, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 702 may also include any memory 706, which is used to store any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 706 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 702. In one case, when the processor 704 executes an associated instruction stored in any memory or combination of memories, the computer device 702 may perform any operation of the associated instruction. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0138] The computer device 702 may also include an input / output module 710 (I / O) for receiving various inputs (via input devices 712) and for providing various outputs (via output devices 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface (GUI) 718. In other embodiments, the input / output module 710 (I / O), the input device 712, and the output device 714 may not be included, and the computer device 702 may be used as a computer device in a network. The computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.

[0139] The communication link 722 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0140] Corresponding to Figures 1 to 5In addition to the method shown, an embodiment of the specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0141] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 5 The method shown.

[0142] The embodiments of the present specification also provide a computer program product, including at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the following Figures 1 to 5 The method shown.

[0143] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0144] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0145] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0149] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0151] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. A method for evaluating the execution of a system, characterized in that: The method comprises: Obtain the content of the target system and the data on the implementation of the target system; Extract keywords from the content of the target system, obtain a list of execution control points of the target system, and determine the weight and monitoring rules of each execution control point in the list of execution control points, wherein the list of execution control points includes conduction learning control points, risk control points, and violation problem control points; Associating the target system execution data with the execution control points in the execution control point list; Determining the quantitative score of the execution control point associated with the target system execution data according to the monitoring rules of the execution control point; Obtaining an execution score of the target system according to the quantitative scores of the execution control points associated with the target system execution data and the weights of the execution control points associated with the target system execution data; The execution evaluation result of the target system is determined according to the execution score of the target system.

2. The method according to claim 1, characterized in that: The control points of conduction learning include training status, system reading volume and material download volume; The risk-type control points include control points of multiple risk levels; The violation control points include internal and external inspection issues, production incidents and negative suggestions on system implementation.

3. The method according to claim 1, characterized in that: The keyword extraction of the target system content to obtain the execution control point list of the target system includes: Extracting a first keyword set of the target system from the target system content using a predefined regular expression; Matching the target system content with a preset associated word list to obtain a second keyword set for the target system, wherein the associated word list is set based on the target system content; Merging the first keyword set and the second keyword set to obtain a third keyword set; De-duplication processing is performed on the third keyword set to obtain an execution control point list of the target system; The execution control point list is input into a pre-trained classification model to obtain the category of each execution control point.

4. The method according to claim 1, characterized in that: The determining the weight of each execution control point in the execution control point list includes: Constructing a judgment matrix according to the relative importance of each execution control point, wherein the judgment matrix includes a single-layer judgment matrix and a multi-layer judgment matrix; Calculate the first maximum eigenvalue and the corresponding first eigenvector of the single-layer judgment matrix; Performing a consistency check on the first eigenvector according to the first maximum eigenvalue, and if the check passes, using the first eigenvector as a first weight vector; Calculate the second largest eigenvalue and the corresponding second eigenvector of the multi-layer judgment matrix; Performing a consistency check on the second eigenvector according to the second maximum eigenvalue, and if the check passes, using the second eigenvector as a second weight vector; The weight of each execution control point is calculated according to the first weight vector and the second weight vector.

5. The method according to claim 1, characterized in that Before extracting keywords from the target system content to obtain the execution control point list of the target system, the method further includes: Split the target system content to obtain standardized texts of chapters, sections, and articles; Keyword recognition is performed on the standardized text, and system entries containing execution control points are screened according to the recognition results, while system entries not containing execution control points are eliminated.

6. The method according to claim 1, characterized in that The associating the target system execution data with the execution control points in the execution control point list includes: Structuring the target system execution data to obtain structured execution data; Performing text analysis on the structured execution data using a text analysis model to obtain a text analysis result; Extract keywords from the text analysis results according to the execution control point list to obtain a keyword list of the execution data; Calculating the matching degree between each keyword in the keyword list and each execution control point in the execution control point list; The execution control point with the highest matching degree for each keyword is used as the associated execution control point of the keyword.

7. A system execution evaluation device, characterized in that: The device comprises: An acquisition module is used to acquire the target system content and target system execution data; An execution control point extraction and weight determination module is used to extract keywords from the target system content, obtain the execution control point list of the target system and determine the weight and monitoring rules of each execution control point in the execution control point list, wherein the execution control point list includes conduction learning control points, risk control points and violation problem control points; An association module, used for associating the target system execution data with the execution control points in the execution control point list; An execution control point score quantification module, used to determine the quantitative score of the execution control point associated with the target system execution data according to the monitoring rules of the execution control point; An execution score calculation module, used for calculating the execution score of the target system according to the quantitative scores of the execution control points associated with the target system execution data and the weights of the execution control points associated with the target system execution data; The execution capability evaluation module is used to determine the execution capability evaluation result of the target system according to the execution score of the target system.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method according to any one of claims 1 to 6.