Work order risk assessment method and device and electronic equipment

Abstract

The invention discloses a work order risk assessment method and device and electronic equipment. The method comprises the steps that work order information of a target work order is acquired, and the work order information comprises a business system related to the target work order and a business influence range of the target work order on a business process; a target risk rule triggered by the work order information is determined, a risk score corresponding to the target risk rule is determined, and the target risk rule is used for extracting risk features in the work order information; and determining the business system, the business influence range and the risk score as influence factors, and determining the risk level of the target work order according to the influence factors. According to the invention, the technical problem that a work order risk identification method adopted in the related technology is difficult to accurately identify and measure the work order risk is solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, and electronic device for assessing work order risks. Background Art

[0002] During the software delivery process, work order systems are crucial infrastructure for ensuring quality. As the scale of software delivery increases and the process becomes more complex, the challenges facing these systems are also increasing. However, existing technologies for managing work order risks primarily rely on manual review or simple rule-based decisions, which can lead to inefficiencies and low accuracy.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, and electronic device for assessing work order risks, so as to at least solve the technical problem that the work order risk identification methods adopted in related technologies are difficult to accurately identify and quantify work order risks.

[0005] According to one aspect of an embodiment of the present application, a method for assessing work order risk is provided, comprising: obtaining work order information of a target work order, wherein the work order information includes the business system involved in the target work order and the business impact scope of the target work order on the business process; determining a target risk rule triggered by the work order information, and determining a risk score corresponding to the target risk rule, wherein the target risk rule is used to extract risk characteristics from the work order information; determining the business system, business impact scope, and risk score as influencing factors, and determining the risk level of the target work order based on the influencing factors.

[0006] In some embodiments of the present application, it also includes: obtaining historical work order data within a preset time period, wherein the historical work order data includes the timestamp of each historical work order; sorting the historical work order data according to the timestamp to obtain continuous time series data, and cutting the time series data into time windows of preset lengths; within each time window, using a prediction model to predict the predicted number of work orders within each time window; and determining a target time window with risk based on the predicted number of work orders and the actual number of work orders within the corresponding time window.

[0007] In some embodiments of the present application, a target time window with risk is determined based on the predicted number of work orders and the actual number of work orders within the corresponding time window, including: determining the difference between the predicted number of work orders and the actual number of work orders within each time window; determining the anomaly score corresponding to each time window based on the difference, wherein the anomaly score is used to quantify the degree of fluctuation of the difference between the predicted number of work orders and the actual number of work orders; and determining the target time window corresponding to the anomaly score.

[0008] In some embodiments of the present application, it also includes: extracting the first business system involved in each work order from the work order sequence data, wherein the work order sequence data is obtained by sorting the historical work order data according to the timestamp; determining the conflicting work orders that conflict with the first business system in the work order sequence data, and determining the second business system that conflicts from the first business system set composed of the first business system; for each second business system, determining the conflict risk score of each conflicting work order based on the time difference between the conflicting work orders, wherein the conflict risk score is used to quantify the degree of influence of the time difference on the conflict risk.

[0009] In some embodiments of the present application, determining a conflicting work order in which a conflict occurs in the first business system in work order sequence data includes: determining a Boolean conflict matrix, wherein the rows and columns in the Boolean conflict matrix are respectively used to represent different historical work orders in the work order sequence data; when the two historical work orders involved in each element in the Boolean conflict matrix correspond to the same first business system and the time difference corresponding to the timestamps of the two historical work orders is less than a time threshold, marking the position corresponding to the element as a valid value, wherein the valid value is used to indicate that a conflict occurs between the two historical work orders corresponding to the element; and determining the work orders corresponding to all positions with valid values ​​in the Boolean conflict matrix as conflicting work orders.

[0010] In some embodiments of the present application, a recognition model is used to extract the work order information of the target work order; the recognition model is trained in the following manner: a first recognition model is used to identify the position and type of the work order information in the training work order data to obtain a first recognition result; the parameters of the first recognition model are adjusted based on the first recognition result and the first label of the training work order data, and a second recognition model is obtained when the first recognition result meets a first preset condition, wherein the first preset condition is that the accuracy of the first recognition model in identifying the position and type of the work order information is greater than or equal to the first preset accuracy; a second recognition model is used to extract the content of the work order information of the training work order data to obtain a second recognition result; the parameters of the second recognition model are adjusted based on the second recognition result and the second label of the training work order data, and a recognition model is obtained when the second recognition result meets a second preset condition, wherein the second preset condition is that the accuracy of the second recognition model in extracting the content of the work order information is greater than or equal to the second preset accuracy.

[0011] In some embodiments of the present application, a risk prediction model is used to determine the target risk rules triggered by the work order information, and to determine the risk scores corresponding to the target risk rules; the risk prediction model is trained in the following manner: a risk rule data set is obtained, wherein the risk rule data set includes structured coded risk rules; a recognition model is used to match the risk characteristics of the work order information of the training work order data with the risk rules to obtain a third recognition result; the parameters of the recognition model are adjusted based on the third recognition result and the rule labels of the training work order data, and an initial risk prediction model is obtained when the third recognition result meets a third preset condition, wherein the third preset condition is that the accuracy of the recognition model in identifying the risk characteristics of the work order information is greater than or equal to the third preset accuracy; the initial risk prediction model is used to predict the risk score corresponding to the training work order data to obtain a predicted risk score; the parameters of the initial risk prediction model are adjusted based on the predicted risk score and the score label of the training work order data, and a risk prediction model is obtained when the predicted risk score meets a fourth preset condition, wherein the fourth preset condition is that the accuracy of the initial risk prediction model in predicting the risk score is greater than or equal to the fourth preset accuracy.

[0012] In some embodiments of the present application, after determining the risk level of the target work order based on the influencing factors, the method also includes: determining a target case that matches the risk level of the target work order from a historical case library, wherein the historical case library is used to store risk assessment records of historical work orders; determining a risk management plan corresponding to the target case, wherein the risk management plan is used to reduce or eliminate the work order risk; and determining a target risk management plan corresponding to the target work order based on the risk management plan and the risk level.

[0013] In some embodiments of the present application, the business system, business impact scope and risk score are determined as influencing factors, and the risk level of the target work order is determined based on the influencing factors, including: determining a first eigenvalue corresponding to the type of the business system and a second eigenvalue corresponding to the business impact scope; obtaining weights corresponding to the first eigenvalue, the second eigenvalue and the risk score, respectively, and determining the target score of the target work order based on the weights; mapping the target score to a preset risk level classification to obtain the risk level.

[0014] According to another aspect of the embodiment of the present application, a work order risk assessment device is also provided, including: an acquisition module for acquiring work order information of a target work order, wherein the work order information includes the business system involved in the target work order and the business impact scope of the target work order on the business process; a scoring module for determining the target risk rules triggered by the work order information, and determining the risk score corresponding to the target risk rules, wherein the target risk rules are used to extract risk characteristics from the work order information; a determination module for determining the business system, business impact scope and risk score as influencing factors, and determining the risk level of the target work order based on the influencing factors.

[0015] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, the memory being used to store program instructions; the processor being connected to the memory and being used to execute the above-mentioned work order risk assessment method.

[0016] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned work order risk assessment method by running the computer program.

[0017] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned work order risk assessment method when executed by a processor.

[0018] In an embodiment of the present application, by comprehensively analyzing influencing factors such as the business system, business impact scope and risk score of the work order, the purpose of multi-dimensional hierarchical assessment of risks is achieved, thereby achieving the technical effect of accurately assessing the risk level of the work order and improving the efficiency of work order risk assessment, and thus solving the technical problem that the work order risk identification method adopted by the relevant technology is difficult to accurately identify and quantify the work order risk. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 This is a hardware structure block diagram of a computer terminal for a work order risk assessment method according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of a method for assessing work order risk according to an embodiment of the present application;

[0022] Figure 3This is a schematic diagram of the overall process of a work order risk assessment method according to an embodiment of the present application;

[0023] Figure 4 This is a flowchart of risk level classification and response strategy generation for a work order risk assessment method according to an embodiment of the present application;

[0024] Figure 5 1 is a flowchart of a comprehensive work order risk assessment method according to an embodiment of the present application;

[0025] Figure 6 It is a structural diagram of a work order risk assessment device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0029] Large Language Model (LLM): A deep learning-based natural language processing model with a large number of parameters and a complex neural network structure capable of processing and understanding large amounts of text data. In this embodiment, the LLM fine-tunes the text features of work orders to improve its understanding of the work order content and the accuracy of work order information extraction.

[0030] Supervised Fine-Tuning (SFT): The process of further training a pre-trained large-scale model using labeled domain-specific datasets to adapt the model to more specific application scenarios. In this embodiment, SFT is used to optimize the LLM model's understanding of the work order text structure and the identification of key fields in the work order information, ensuring that the model can accurately parse and determine the risk characteristics of the work order.

[0031] Time Series Analysis: A statistical analysis method used to analyze chronologically ordered datasets to identify trends, periodicity, and anomalies. In this embodiment, time series analysis is used to monitor changes in the number of work orders and risk profiles over time. This allows for the identification of peak work order periods and potential system risks, providing data support for risk warnings and strategy adjustments.

[0032] Conflict Detection Algorithm: This algorithm is used to detect and analyze conflicts between two or more operations to ensure system stability and data consistency. In this embodiment, a conflict detection algorithm is used to analyze the risk of overlapping work order operations within a short period of time. By detecting conflicts in operations on the same system at the same time, it provides the operations team with recommendations for adjusting work order execution plans.

[0033] Multifactor Analysis: A method for comprehensively evaluating a complex problem by considering multiple influencing factors. In the embodiments of this application, multifactor analysis is used to comprehensively evaluate the risk level of work orders. By considering multiple factors such as system criticality, business impact, and triggered risk rules, this provides a comprehensive perspective for quantifying and grading work order risks, ensuring that risk response strategies are formulated more rationally and effectively.

[0034] The ticket risk management approaches employed by existing technologies have exposed a series of technical issues and limitations in the face of increasingly complex IT operations environments. These limitations are primarily manifested in the following aspects: First, in traditional ticket processing, risk assessment and decision-making often rely on the experience and intuition of operations personnel. This manual review approach is not only time-consuming and labor-intensive, but also susceptible to individual differences, leading to subjectivity and uncertainty in risk identification, making it difficult to ensure consistency and accuracy. Second, ticket risk assessments are often based on a predefined, relatively static rule base. Due to the rigidity and singleness of these rules, they often fail to address the diversity and complexity of ticket content and struggle to capture unexpected risk factors. Furthermore, existing technologies lack effective mechanisms to monitor and control the risks of concentrated operations on multiple tickets in a short period of time. These operations can lead to excessive consumption of system resources, degraded system performance, or even failures. This risk is particularly significant when it comes to core systems or sensitive business modules. Furthermore, existing technologies typically analyze risk only on a single ticket, ignoring the time series and interdependencies between tickets. This analysis approach fails to comprehensively assess the overall risk profile of the system within a short time window.

[0035] In order to solve the above technical problems, the embodiments of the present application provide corresponding solutions, which are described in detail below.

[0036] The work order risk assessment method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal for implementing a work order risk assessment method is shown in FIG. Figure 1 As shown, the computer terminal 10 may include one or more (illustrated by 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0037] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0038] Memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the work order risk assessment method in the embodiments of the present application. The processor executes the software programs and modules stored in memory 104 to perform various functional applications and data processing, thereby implementing the above-mentioned work order risk assessment method. Memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 104 may further include memory remotely located relative to the processor, and these remote memories may be connected to computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0039] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0040] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0041] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0042] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for assessing work order risks. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Figure 2 This is a flow chart of a method for assessing work order risk according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0044] Step S202 : Acquire the work order information of the target work order, wherein the work order information includes the business system involved in the target work order and the business impact scope of the target work order on the business process.

[0045] In the above step S202, the target work order refers to the operation and maintenance work order currently being processed or analyzed, which is a specific object in the entire risk management process and involves the need for changes, maintenance or other IT operations. Work order information refers to all descriptive data contained in the work order, including but not limited to the title, description, business systems involved, business impact scope, operation type, expected execution time, etc. of the work order. Business systems refer to the core IT systems or applications of an enterprise, which directly support and implement the main business processes of the enterprise, such as payment processing systems, user authentication systems, log monitoring systems, etc. The scope of business impact refers to the scope and degree of impact that the work order operation may have on the business process, service quality or user experience, covering the business lines, functional modules, user groups that the operation may affect, and the possible business losses or inconveniences.

[0046] Specifically, you can parse the text content of the target work order to extract fields containing key information such as the business system and business impact range. For example, you can use the following two methods:

[0047] (1) Rule-based extraction: Design a series of regular expressions or keyword rules to identify patterns in the work order text that describe the business system and business impact scope.

[0048] (2) Machine learning-based recognition: Utilize the trained named entity recognition (NER) model to automatically identify and classify entity information in the work order, such as system name, operation type, etc.

[0049] In some embodiments of the present application, a recognition model can be used to extract the work order information of the target work order; the recognition model is trained in the following manner: a first recognition model is used to identify the position and type of the work order information in the training work order data to obtain a first recognition result; the parameters of the first recognition model are adjusted based on the first recognition result and the first label of the training work order data, and a second recognition model is obtained when the first recognition result meets a first preset condition, wherein the first preset condition is that the accuracy of the first recognition model in identifying the position and type of the work order information is greater than or equal to the first preset accuracy; a second recognition model is used to extract the content of the work order information of the training work order data to obtain a second recognition result; the parameters of the second recognition model are adjusted based on the second recognition result and the second label of the training work order data, and a recognition model is obtained when the second recognition result meets a second preset condition, wherein the second preset condition is that the accuracy of the second recognition model in extracting the content of the work order information is greater than or equal to the second preset accuracy.

[0050] The first recognition model is an initial model, primarily used to identify the location and type of work order information. Its role is to preliminarily parse the work order text, locate the locations of key fields, and categorize their types, such as marking the locations of "system name," "operation time," and other content. Specifically, a deep learning platform such as TensorFlow or PyTorch can be used to train a sequence annotation-based model, such as BiLSTM-CRF, to identify the location and type of work order information. Historical work order data is manually annotated to mark the locations and types of key fields, and then input into the first recognition model as a training set. Model parameters are adjusted through multiple rounds of iterative training until the model accuracy reaches a first preset accuracy level.

[0051] The second recognition model is an advanced model, optimized based on the first recognition model. It focuses on accurately extracting the content of work order information, ensuring the accuracy and completeness of the information extracted. Specifically, based on the first recognition model, the model parameters are continuously adjusted to optimize the accuracy of work order information location and type recognition until the first preset condition is met. For example, some parameters and architecture of the optimized first recognition model can be transferred to the second recognition model, retaining the location and type recognition capabilities, but focusing on the model's accurate content extraction.

[0052] For the training of the second recognition model, at the work order information location identified by the first recognition model, the second recognition model focuses on extracting specific content, such as the specific value of the system name, the specific date and time of the operation time, etc. For example, a work order data set with content labels is used to supervise the training, and the parameters of the second recognition model are adjusted until the model's extraction accuracy of the work order information content meets the second preset condition, thereby obtaining a recognition model.

[0053] Step S204: determining the target risk rule triggered by the work order information and determining the risk score corresponding to the target risk rule, wherein the target risk rule is used to extract risk features from the work order information.

[0054] In step S204 above, target risk rules refer to specific criteria or conditions used to identify potential risk characteristics in work order information. Risk rules can be developed based on historical data and operational experience, aiming to identify security, stability, and compliance issues that may arise from work order operations. For example, risk rules may include system changes during peak business periods, operational processes involving sensitive data, and operation frequencies exceeding a certain threshold.

[0055] The risk score is a numerical value that quantifies the risk level of a work order based on the risk rules triggered by the target work order. A higher risk score indicates a greater risk for the work order operation.

[0056] For example, the key fields extracted from the work order information are matched with the predefined risk rule library, and keyword search or rule engine is used to determine whether the work order triggers a specific risk rule; once the work order matches the rule, the relevant risk features in the work order are identified and extracted, such as the business system involved, sensitive operations, operation time points, etc.; it should be noted that a weight value can be set for each risk rule to indicate the importance of the rule in the overall risk assessment. For example, the rule weight of a high-risk operation instruction may be higher than the rule weight of a system name; the risk score is calculated based on the rules triggered by the work order and the rule weights.

[0057] In some embodiments of the present application, a risk prediction model can be used to determine the target risk rules triggered by the work order information, and to determine the risk score corresponding to the target risk rule; the risk prediction model is trained in the following manner: obtaining a risk rule data set, wherein the risk rule data set includes structured coded risk rules; using a recognition model to match the risk characteristics of the work order information of the training work order data with the risk rules to obtain a third recognition result; adjusting the parameters of the recognition model based on the third recognition result and the rule labels of the training work order data, and obtaining an initial risk prediction model when the third recognition result meets a third preset condition, wherein the third preset condition is that the accuracy of the recognition model in identifying the risk characteristics of the work order information is greater than or equal to the third preset accuracy; using the initial risk prediction model to predict the risk score corresponding to the training work order data to obtain a predicted risk score; adjusting the parameters of the initial risk prediction model based on the predicted risk score and the score label of the training work order data, and obtaining a risk prediction model when the predicted risk score meets a fourth preset condition, wherein the fourth preset condition is that the accuracy of the initial risk prediction model in predicting the risk score is greater than or equal to the fourth preset accuracy.

[0058] The risk prediction model is a machine learning model specifically used to predict the risk level of operation and maintenance operations based on work order information. It is trained based on historical work order data and risk rule data sets. It can automatically identify risk features in work order information and predict risk scores accordingly.

[0059] The risk rule dataset contains a collection of various risk rules. These rules represent common risk types and indicators in the IT operations field and have been structured and encoded for easy model understanding and utilization. For example, they include definitions of sensitive operations, limits on peak business hours, and criteria for determining system criticality. Structured encoding refers to the process of converting risk rules into numerical or vector form that the model can understand.

[0060] Specifically, to facilitate understanding of the training process of the above-mentioned identification model and risk prediction model, the following is explained in conjunction with some specific embodiments, including:

[0061] S1: Historical data preprocessing.

[0062] (1) Data collection and classification.

[0063] The system obtains and collects basic work order information and work order risk rule data from the historical work order database. The basic work order information includes: work order title, description, timestamp, system name and impact scope, etc., which is used by the model to understand the structure and content of the work order; the work order risk rules include change time limits (for example: change limits during peak periods), high-risk operation instructions, etc., which are used for subsequent risk assessment and rule extraction.

[0064] (2) Data cleaning and standardization.

[0065] Clean the noise data in the work order text, such as removing redundant characters and irrelevant tags, and use regular expression matching technology to standardize fields such as timestamps, system names, and business impact scopes.

[0066] For example, to convert different time formats into a unified timestamp format, the data cleaning formula is as follows:

[0067] Standardized_Timestamp=convert_to_timestamp(original_timestamp)(Formula 1)

[0068] Here, original_timestamp is the original timestamp, and Standardized_Timestamp is the standardized timestamp.

[0069] It should be noted that for the impact scope field, you can use the mapping table between business systems and impact labels to uniformly map custom names to standard business field classifications.

[0070] (3) Structured coding of risk rules.

[0071] The system encodes risk rules so that they match the input format of subsequent models. Encoding methods include, but are not limited to: marking the change time window limit with a Boolean variable (1 for restricted time, 0 for unlimited time); encoding high-risk operation instructions as a fixed keyword matching list.

[0072] It should be noted that the keyword table can be used to construct word vectors to improve the recognition rate of risk rules during model training.

[0073] S2: Fine-tune the work order content understanding model.

[0074] (1) Data selection and model fine-tuning.

[0075] Using cleaned and standardized ticket data (i.e., training ticket data), we fine-tune the Large Language Model (LLM). This process can employ supervised fine-tuning (SFT). The training goal is to enable the model to parse the ticket text structure and identify key information in fields such as titles and descriptions.

[0076] (2) Fine-tuning method.

[0077] Select high-quality work order samples with labels (for example, you can filter data that meets preset conditions from the training work order data to form high-quality work order samples), fine-tune the pre-trained language understanding ability of the LLM model, and optimize the model's ability to parse work order content.

[0078] For key field extraction, the NLP word vector generation method and named entity recognition (NER) algorithm can be combined to extract key information in the text (such as system name, time, change content, etc.).

[0079] The formula is as follows:

[0080] Key_Information=NER(Input_Text) (Formula 2)

[0081] Among them, the NER algorithm maps the field information (Input_Text) to the field label (Key_Information) required by the system, enhancing the accuracy of the model's analysis of work order information.

[0082] S3: Fine-tuning of risk identification model.

[0083] (1) Introduce work order risk rule data.

[0084] Based on S2, the system introduces work order risk rules and further fine-tunes the annotation of risk information in combination with the LLM model. The fine-tuning goal is to enable the model to accurately match the potential risk characteristics in the work order and perform risk assessment based on the rule base.

[0085] (2) Risk rule mapping and feature extraction.

[0086] The system uses a deep learning algorithm based on the attention mechanism to identify risk rules triggered in work orders. For example, it can analyze the similarity between the text content of the work order and the risk rules and use a convolutional neural network (CNN) for feature extraction. The formula is as follows:

[0087]

[0088] Among them, α i is the weight coefficient of each risk feature, f is the feature mapping function, and Risk_Score is the calculated work order risk score.

[0089] Step S206 : Determine the business system, business impact scope, and risk score as impact factors, and determine the risk level of the target work order based on the impact factors.

[0090] In the above step S206, the following steps can be used to determine the risk level of the target work order: determine a first characteristic value corresponding to the type of business system and a second characteristic value corresponding to the business impact scope; obtain weights corresponding to the first characteristic value, the second characteristic value and the risk score, respectively, and determine the target score of the target work order based on the weights; map the target score to a preset risk level classification to obtain the risk level.

[0091] Specifically, the system uses a multi-factor analysis method to classify the risk of work orders based on system criticality, business impact scope, and triggered risk rules. The target score calculation formula is as follows:

[0092] Risk_Level=Weight system ·S system +Weight impact ·S impact +Weight rule ·S rule (Formula 4)

[0093] Among them, Weight system 、Weight impact 、Weight rule Represents the weight of each impact factor, S system 、S impact 、S ruleThey represent the first eigenvalue, the second eigenvalue, and the risk score (i.e., the score corresponding to the triggered risk rule) respectively.

[0094] After determining the risk level of the target work order based on the influencing factors, the following steps can also be performed: determine a target case that matches the risk level of the target work order from the historical case library, where the historical case library is used to store risk assessment records of historical work orders; determine the risk management plan corresponding to the target case, where the risk management plan is used to reduce or eliminate the work order risk; determine the target risk management plan corresponding to the target work order based on the risk management plan and the risk level.

[0095] Specifically, the system automatically generates corresponding response suggestions from a policy library based on risk level. For example, low-risk work orders automatically generate handling suggestions, medium-risk work orders prompt additional approvals, and high-risk work orders require a comprehensive review. This dynamic generation mechanism analyzes key attributes of high-risk work orders (such as time, script, and approval flow) based on real-time work order data. It then draws on historical success cases to generate personalized reference solutions, enhancing the relevance of response suggestions.

[0096] In some embodiments of the present application, the following steps can also be performed: obtaining historical work order data within a preset time period, wherein the historical work order data includes the timestamp of each historical work order; sorting the historical work order data according to the timestamp to obtain continuous time series data, and cutting the time series data into time windows of preset lengths; within each time window, using a prediction model to predict the predicted number of work orders within each time window; determining a target time window with risk based on the predicted number of work orders and the actual number of work orders within the corresponding time window.

[0097] A preset time period refers to the period of historical data collection selected for ticket volume forecasting and risk assessment, such as a week, month, or specific maintenance cycle. The timestamp is a key field in historical ticket data, indicating the specific time when the ticket was submitted or expected to be executed. Time series data consists of historical ticket data sorted by timestamp, forming a data series that changes over time.

[0098] Determine the target time window with risk based on the predicted number of work orders and the actual number of work orders within the corresponding time window: In each time window, determine the difference between the predicted number of work orders and the actual number of work orders; determine the anomaly score corresponding to each time window based on the difference, where the anomaly score is used to quantify the degree of fluctuation in the difference between the predicted number of work orders and the actual number of work orders; and determine the target time window corresponding to the anomaly score.

[0099] Specifically, the system uses time series analysis to perform a longitudinal comparison of work order data from different time periods to identify abnormal situations (such as a surge in work orders, frequent system changes, etc.). The time series formula is as follows:

[0100]

[0101] Among them, Observed_Count is the number of work orders in the current time period, Expected_Count is the predicted number of work orders, σ count is the standard deviation. If the Anomaly_Score (abnormal score) exceeds the set threshold, the system triggers an early warning.

[0102] In some embodiments of the present application, the following steps can also be performed: extracting the first business system involved in each work order from the work order sequence data, wherein the work order sequence data is obtained by sorting the historical work order data according to timestamps; determining the conflicting work orders that conflict with the first business system in the work order sequence data, and determining the second business system that conflicts from the first business system set composed of the first business system; for each second business system, determining the conflict risk score of each conflicting work order based on the time difference between the conflicting work orders, wherein the conflict risk score is used to quantify the degree of influence of the time difference on the conflict risk.

[0103] Conflicting work orders are work orders that are in the same period or system, have similar operation times, and may affect each other. The first business system set is a set consisting of business systems that appear in all historical work order data.

[0104] For example, you can traverse the work order sequence data, extract the system name field of each work order, and determine the first business system involved in each work order based on the system classification mapping table; use the Boolean conflict matrix (Conflict Matrix) and time difference threshold to identify conflicting work orders. If the operation time difference between two work orders on the same system or involving dependent systems is less than a certain threshold, they are considered to be in conflict.

[0105] The conflicting work orders that conflict with the first business system in the work order sequence data can be determined by the following steps: determining a Boolean conflict matrix, wherein the rows and columns in the Boolean conflict matrix are respectively used to represent different historical work orders in the work order sequence data; when the two historical work orders involved in each element in the Boolean conflict matrix correspond to the same first business system and the time difference corresponding to the timestamps of the two historical work orders is less than a time threshold, marking the position corresponding to the element as a valid value, wherein the valid value is used to indicate that the two historical work orders corresponding to the element conflict; and determining the work orders corresponding to all positions with valid values ​​in the Boolean conflict matrix as conflicting work orders.

[0106] The Boolean conflict matrix is ​​a two-dimensional array structure used to represent the conflict status between each pair of work orders in the work order sequence data. Each element in the matrix indicates whether there is a conflict between a pair of work orders. For example, 1 indicates a conflict (i.e., a valid value) and 0 indicates no conflict.

[0107] Specifically, an n×n zero matrix can be initialized, where n is the number of work orders in the work order sequence data; the work order sequence data is traversed, and for any two work orders i and j, if i and j involve the same first business system and the difference between their timestamps is less than a preset time threshold, then the (i, j)th position in the Boolean conflict matrix is ​​marked as 1, otherwise it is marked as 0; the Boolean conflict matrix is ​​analyzed to find all elements marked as 1, which correspond to conflicting work order pairs; all work orders marked as 1 in the Boolean conflict matrix are collected to form a conflicting work order set.

[0108] Specifically, the system uses a conflict detection algorithm to analyze work orders that are concentrated in a short period of time to identify the overlapping risks of the same system. For example, the system uses a Boolean conflict matrix to determine whether multiple work orders are concentrated in the same system and time period. The conflict detection formula is:

[0109]

[0110] Among them, System i and System j Represents the business systems of two historical work orders respectively, indicating |t i -t j |The time difference of the work order timestamp, Δt is the time threshold for conflict determination.

[0111] It's important to note that after the assessment is complete, the system generates a risk warning report based on the results of risk identification and conflict detection. This report includes the distribution of high-risk tickets, the risk situation within a specific time period, and recommendations for addressing multi-ticket risks. This report supports the operations team in optimizing change windows and system protection strategies.

[0112] Through the above steps S202 to S206, by comprehensively analyzing the influencing factors such as the business system, business impact scope and risk score of the work order, the purpose of multi-dimensional hierarchical assessment of risks is achieved, thereby achieving the technical effect of accurately assessing the risk level of the work order and improving the efficiency of work order risk assessment, and thus solving the technical problem that the work order risk identification method adopted by the relevant technology is difficult to accurately identify and quantify the work order risk.

[0113] In order to facilitate understanding of the above technical solution, some specific embodiments are explained below.

[0114] In a simulated enterprise IT operation and maintenance environment, involving multiple core subsystems and multiple types of operations (configuration changes, code releases, script executions, data migrations, etc.), in order to ensure system stability and efficient operation and maintenance, the operation and maintenance team needs to accurately identify risks and dynamically control complex work order operations. Work order operations include routine changes, script executions, emergency releases, etc. To this end, a rich set of risk rules is introduced in the embodiment of the present application, and LLM large model training and learning are used to identify potential risk operations of different work orders, evaluate the risks of multiple work orders, and generate risk levels and corresponding dynamic response strategies in real time. The specific technical solution may include the following steps:

[0115] S1: Historical data preprocessing: Collect and clean historical work order data, which is mainly divided into basic work order information (title, description, timestamp, system name, impact scope, etc.) and work order risk rules (such as change time limit, prohibited operations and other risk criteria).

[0116] S2: Fine-tuning the work order content understanding model: Using clear, accurate, and high-quality work order data, fine-tune the LLM model so that it can accurately parse the work order text, identify key information, and judge the clarity and accuracy of the work order content, providing support for subsequent risk identification.

[0117] S3: Fine-tuning the risk identification model: Based on the understanding of the work order content, the work order risk rule data is introduced and the LLM model is fine-tuned a second time to enable it to intelligently identify potential risks in the work order. The model extracts risk features according to the rules and performs risk assessment and identification on the work order.

[0118] S4: Risk Classification and Response Strategy Generation: Based on the identification of work order risks, multi-factor analysis is used to classify risks (low, medium, high, etc.), and a corresponding response strategy library is generated based on different risk levels. For example, processing suggestions are automatically generated for low-risk work orders, and optimization suggestions are provided for medium- and high-risk work orders, such as adjusting operation time and increasing approvals, to effectively reduce risks.

[0119] S5: Comprehensive Work Order Risk Assessment: Through time series and conflict detection analysis, we conduct a comprehensive assessment of the risks associated with multiple work orders, identifying system risks that may arise from multiple work order operations in the short term. Based on the assessment results, we generate a risk warning report to facilitate timely adjustments to O&M strategies and optimize change windows to prevent system stability issues.

[0120] Specifically:

[0121] (1)Environmental settings.

[0122] System structure: The enterprise system includes payment processing module, user authentication module, log monitoring system, data storage system and content delivery network (CDN).

[0123] Work Order Description: 10 work orders, covering operations such as configuration updates, code releases, and batch script execution. Each work order contains specific operation content and its potential impact.

[0124] Risk rule base: A set of risk rule bases is preset, such as prohibiting large-scale updates during critical periods, requiring a second review of payment system changes, and considering work orders that involve core systems multiple times as high risk.

[0125] (2) Data preparation.

[0126] Basic work order information: includes title, description, timestamp, involved subsystems, business impact scope, summary of change content, expected execution time, etc.

[0127] Work Order 1: Payment system configuration adjustment (security configuration update of the payment gateway module, involving sensitive interface settings), 2024-11-15 09:00.

[0128] Work Order 2: Log Monitoring System Upgrade (update the monitoring system version), 2024-11-15 09:05.

[0129] Work Order 3: Database Migration (transaction database data migration to the new cluster), 2024-11-15 09:10.

[0130] Ticket 4: User authentication module update (optimized authentication logic), 2024-11-15 09:15.

[0131] Work Order 5: Traffic Management System Configuration Change (Load Balancing Parameter Adjustment), 2024-11-15 09:20.

[0132] Work Order 6: Payment System Security Patch (Payment System Security Vulnerability Fix), 2024-11-15 09:25.

[0133] Work Order 7: Data storage module restart (data cleanup operation requires a brief system shutdown), 2024-11-15 09:30.

[0134] Ticket 8: Payment gateway monitoring function optimization (increase monitoring frequency), 2024-11-15 09:35.

[0135] Work Order 9: Data Storage Configuration Update (New Backup Policy), 2024-11-15 09:40.

[0136] Work Ticket 10: Real-time trading system load test (testing the trading module's carrying capacity), 2024-11-15 09:45.

[0137] Some risk rules may specifically include:

[0138] Rule 1. High-risk system rule: All work orders involving payment systems or user verification modules are considered high-risk and require a second review.

[0139] Rule 2. Time restrictions: All system changes are prohibited between 00:00 and 06:00 daily. Peak business hours are also restricted: If the execution time conflicts with peak business hours, all high-load or sensitive operations must be delayed or rescheduled.

[0140] Rule 3. Operation frequency rule: a maximum of two work order operations are allowed on the same system per hour. If the limit is exceeded, a prompt will be displayed and rescheduling will be recommended.

[0141] Rule 4. Multi-ticket risk assessment: Changes affecting multiple core systems (payment, verification, storage, and monitoring) during the same period require an assessment of the risk of overlap and appropriate mitigation measures. Simultaneous high-load operations on two or more systems will trigger an alert, and it is recommended that the operation be timed appropriately to minimize the impact of system load.

[0142] Rule 5. Code and script risk rule: Code or scripts containing high-risk instructions, unauthorized scripts, and database operations (such as file deletion and root permission modification) are all marked as high-risk operations.

[0143] Rule 6. System Interdependency Risk Rule: Multiple work orders involving different modules within the same business chain (e.g., payment processing and user authentication) require an assessment of the impact of these modules' interactions. If more than two work orders involve the same business chain, the risk of cumulative impact must be calculated and non-essential work orders must be delayed.

[0144] Rule 7. Conflict detection rule: When more than two batch operations or scripts are executed on the same system module within the same period, it is considered high risk and triggers conflict detection and assessment.

[0145] Rule 8. Historical failure record rule: Based on historical data analysis, if the failure rate of similar operations in the past month exceeds 30%, the risk weight of the current work order will be automatically increased and marked as high risk.

[0146] Rule 9. System load monitoring rule: If the current system load is higher than 70%, all work orders except necessary ones must be delayed to avoid affecting system stability.

[0147] Risk rule weight allocation:

[0148] Rule 1: High-risk system rule—20 points.

[0149] Rule 2: Time limit rule - 15 minutes.

[0150] Rule 3: Operation frequency rule - 10 points.

[0151] Rule 4: Risk of overlapping multiple work orders - 15 points.

[0152] Rule 5: Code and Script Risks – 20 points.

[0153] Rule 6: System Interdependence Risk - 15 points.

[0154] Rule 7: Conflict Detection Rules – 10 points.

[0155] Rule 8: Historical failure record reference - 15 points.

[0156] Rule 9: System load monitoring - 10 points.

[0157] (3) Implementation steps.

[0158] Intelligently classify work order risk levels, analyze work order content based on the LLM model, and identify the impact scope of the operation type (code release, configuration adjustment, script execution), operating system, and business chain.

[0159] The work orders and their risk scores are shown in Table 1:

[0160]

[0161]

[0162]

[0163] Intelligently generate response strategies and classify work orders into three risk levels: low, medium, and high based on the calculated scores. Among them, work orders related to payment modules, script releases, and database operations are prioritized as high risk.

[0164] Low risk (0-30 points): Execute normally, no additional review required.

[0165] Medium risk (31-60 points): It is recommended to increase the review or adjust the execution time.

[0166] High risk (61-90 points): requires a second review and may be adjusted to off-peak hours.

[0167] Extremely high risk (91 points and above): Full review, delayed execution, and notification to the responsible business person.

[0168] Based on the grading standards set above and the risk scores of each work order, an intelligent response strategy is generated as shown in Table 2 below:

[0169]

[0170] 1) Comprehensive assessment of the risks of multiple work orders.

[0171] Risk score breakdown and calculation:

[0172] Conflict detection weight W c =5;

[0173] High-risk operation weight W h =15;

[0174] Dependence on risk weight W d =10;

[0175] Load risk weight W i =20.

[0176] Specific work order risk calculation:

[0177] Conflict risk score: For work orders with more than 2 instances in a single system, the conflict risk score = (n-2) × W c .

[0178] High-risk operation score: Each high-risk operation (such as database operation, unauthorized script) is scored W h point.

[0179] Dependency risk score: Cross-module dependencies lead to risk propagation between systems, and the work order dependency score is W. d .

[0180] Load risk score: When the current system load is > 70%, the excess load is multiplied by the weight (L-70) × W. i calculate.

[0181] The work order scoring table and calculation details are shown in Table 3:

[0182]

[0183]

[0184] 2) Detailed calculation examples for each dimension.

[0185] Work Order 1:

[0186] Conflict detection: The payment system module conflicts with work orders 6 and 8 (3 times), exceeding the limit by 1 time. Conflict score = 1 × 5 = 5.

[0187] High-risk operations: 15 points for operations involving sensitive interfaces of the payment system (high-risk operations).

[0188] Dependency Risk: The payment module has a dependency on the authentication module, with a score of 10.

[0189] Load risk: None (no high load currently).

[0190] Total score: 5+15+10=35.

[0191] Work Order 6:

[0192] Conflict detection: The payment system module conflicts with work orders 1 and 8 (3 times), exceeding the limit by 1 time. Conflict score = 1 × 5 = 5.

[0193] High-risk operations: 15 points for operations involving payment system security patches (high-risk operations).

[0194] Dependency risk: None.

[0195] Load risk: None.

[0196] Total score: 5+15=25.

[0197] Work Order 10:

[0198] Conflict detection: None.

[0199] High-risk operations: None.

[0200] Dependency risk: None.

[0201] Load risk: The system load currently exceeds 70%, load score = (75-70)×20=20.

[0202] Total score: 20.

[0203] Figure 3 This is a schematic diagram of the overall process of a work order risk assessment method according to an embodiment of the present application. Figure 3 As shown, it includes the following steps:

[0204] Step S302: historical data preprocessing.

[0205] Collection and Classification: All work order records are collected from the historical database of the enterprise IT operations system, including title, description, timestamp, involved business system, operation type, change script, and other content. At the same time, work order types are categorized, such as configuration changes, code deployments, and database operations, as well as work order risk rules, such as operation restrictions during sensitive periods, operation frequency limits, and system load thresholds.

[0206] Data cleaning and standardization: De-noising and standardization are performed on the collected data. For example, regular expressions are used to remove redundant information such as HTML tags and non-printable characters from work order descriptions; timestamps are converted to a standard format to facilitate time series analysis; and information such as system names and business areas are standardized and encoded to ensure consistency and comparability.

[0207] Standardized work order data: After cleaning and standardization, the work order data will form a structured dataset in which the field information of each work order is complete and unified, providing high-quality input data for subsequent model fine-tuning and risk assessment.

[0208] Work order risk data: Organize all risk data related to work order operations, including historical work order processing results, risk event records, system health status, etc., as the data basis for subsequent risk prediction and assessment.

[0209] Step S304: Fine-tune the work order content understanding model.

[0210] A portion of historical work order data is selected as the training set. This portion of work order data should be clear, accurate, and successfully processed. Using this high-quality data, supervised fine-tuning of the Large Language Model (LLM) is performed to enhance the model's understanding of work order text, enabling it to accurately identify key information such as system name, operation time, and change content. A pre-trained large language model, such as GPT-3 and BERT, is used as the base model. Through SFT, the model can learn specific patterns in work order data, improving the accuracy and efficiency of work order parsing.

[0211] Step S306: fine-tuning the risk identification model.

[0212] Based on historical ticket risk data and ticket risk rules, the Large Language Model (LLM) underwent a second fine-tuning exercise, focusing on model learning and identifying potential risks within ticket operations. This process enabled the model to understand which characteristics, such as operation types, time periods, and system connections, may indicate higher risk. Similar to the ticket content understanding model, the large prediction model here refers to the model that, after the first fine-tuning, has a better understanding of ticket text. Further SFT training enhanced the model's risk identification capabilities, enabling it to more reliably assess potential risks within ticket operations.

[0213] Step S308: Risk level classification and response strategy generation.

[0214] Multi-factor analysis: This method comprehensively considers multiple dimensions, such as the time point of the work order operation, the criticality of the business system, the operation type, and the historical success rate, to conduct a multi-factor analysis of the work order.

[0215] Determine risk classification: Based on the results of multi-factor analysis, work orders are divided into four levels: low risk, medium risk, high risk, and extremely high risk. Each level corresponds to a different risk handling process and approval mechanism.

[0216] Generate response strategies: For work orders with different risk levels, the system automatically generates corresponding response strategies. For example, low-risk work orders can be automatically executed, medium-risk work orders require additional approval or review, and high-risk work orders require a comprehensive review, which may include delaying execution or modifying the action plan. Extremely high-risk work orders may require a special meeting for discussion.

[0217] Specifically, Figure 4FIG. 1 is a flow chart of risk level classification and response strategy generation according to a work order risk assessment method according to an embodiment of the present application. Figure 4 As shown, it includes the following steps:

[0218] (1) Quantitative analysis of risk factors.

[0219] System Criticality: Each business system is scored based on its importance in the enterprise IT architecture and its impact on business continuity. Systems with high criticality are weighted higher because any changes to them could cause major business disruptions.

[0220] Business impact scope: Assess the potential impact of work order operations on business processes and customer experience. The wider the impact scope, the greater the potential risk to the business.

[0221] Work Order Risk Score: Based on model prediction, combined with work order content and historical data, each work order operation is assigned a risk score. The higher the score, the greater the potential risk of the work order operation.

[0222] Risk rules for penalties: Taking into account specific business rules or historical risk events, additional penalties are imposed on work order risks, such as performing high-risk operations during business peak periods.

[0223] (2) Quantitative calculation formula for risk level.

[0224] The risk level can be quantitatively calculated with reference to Formula 4 in the above embodiment.

[0225] (3) Risk level classification.

[0226] Risk classification is based on the calculated comprehensive risk score, and the work order is divided into four levels: low risk, medium risk, high risk, and very high risk. For example:

[0227] Low Risk: The risk score is between 0 and 30. The work order operation has little impact on the system and business and can be executed according to standard procedures.

[0228] Medium risk: Risk scores range from 31 to 60. Work order operations require additional attention and may affect system stability or business continuity.

[0229] High Risk: Risk scores range from 61 to 90. This indicates that the work order operation carries a high risk and requires a thorough review, which may require adjusting execution times or implementing additional security measures.

[0230] Very High Risk: A risk score of 90 or more indicates that the work order operation is extremely dangerous and could cause system failure or major business interruption. The work order must be suspended for a comprehensive assessment and notified to senior management.

[0231] (4) Generate personalized reference plans by combining level classification, real-time work orders and historical success cases.

[0232] Real-time work order data: The system receives newly submitted work orders in real time, performs risk assessment and classifies them into different levels through the model.

[0233] Historical success cases: Extract the processing procedures and results of similar-level work orders from historical work order data as reference cases.

[0234] Personalized solution generation: Based on the risk level of the current work order, the system automatically retrieves historical successful cases for handling strategies for similar work orders and generates a personalized reference solution based on the specific context of the current work order. For example, for medium-risk work orders, the system may recommend adjusting the execution time to off-peak business periods or adding additional approval processes, while high-risk work orders may require comprehensive review, testing, and the preparation of contingency plans.

[0235] (5) Generation of specific response strategies.

[0236] Low-risk work orders: Automatically generate handling suggestions, such as automated execution, without the need for additional approval, to speed up work order processing.

[0237] Medium-risk work orders: An approval process is added, requiring approval from higher-level operation and maintenance management personnel to confirm the rationality and safety of the change plan.

[0238] High-risk tickets require a comprehensive review, including but not limited to script code review, business impact assessment, and system stability testing. After passing the review, the change window is adjusted based on the risk level to ensure that the operation is carried out during a period of low system load.

[0239] Step S310: Comprehensive assessment of work order risks.

[0240] Time series analysis: Analyze the sequence of work order submission times to identify whether there is a surge or peak in work orders. Through time series analysis, the operation and maintenance team can predict future work order loads and prepare necessary resources in advance.

[0241] Conflict detection analysis: Construct a Boolean conflict matrix to check whether there are operational conflicts in the work order sequence involving the same business system or related systems, especially for work orders that are close in time.

[0242] Generate risk warning reports: Based on the results of time series analysis and conflict detection, the system generates risk warning reports that list high-risk work orders, conflicting work orders, system load forecasts, and recommended response strategies, providing the operation and maintenance team with a detailed action guide to prevent and respond to potential risks in advance.

[0243] Specifically, Figure 5FIG. 1 is a flow chart of a comprehensive work order risk assessment method according to an embodiment of the present application, such as Figure 5 As shown, it includes the following steps:

[0244] (1) Obtain historical work order cases.

[0245] Extract and summarize historical work order data from the enterprise's IT operation and maintenance database, including but not limited to detailed operation descriptions of the work order, the systems involved, the operation time, the scope of business impact, and the final processing results. This data constitutes the work order case library, which is an important basis for subsequent time series analysis and multi-work order conflict detection.

[0246] (2) Time series analysis.

[0247] Identifying a surge in work orders: By analyzing the number of work orders in a time series, we can identify any abnormal increase in the number of work order submissions, which may indicate that the system is about to be under high load or that the operations team is under increased pressure.

[0248] Frequent system change detection: Monitors the number of changes to a specific system within a short period of time. If the detected change frequency is higher than the historical average, the system may be unstable and pose a high risk.

[0249] Abnormal situation verification: Use anomaly detection algorithms (such as Z-score, IQR method, or machine learning-based anomaly detection models) to analyze the changing trends of the number of work orders and system load, identify behaviors that deviate from the normal range, and thus provide early warning of potential risks.

[0250] Trigger an alert: If any of the above analysis results indicate an abnormal situation, the system will immediately trigger an alert to notify the operation and maintenance team to pay attention to these potential risks.

[0251] (3) Multiple work order conflict detection.

[0252] Identification of short-term batch work orders: Analyze multiple work orders executed on the same system or related systems within a short period of time to assess their impact on system stability.

[0253] Boolean conflict matrix detection: Construct a matrix where the rows and columns represent all work orders in the work order sequence. The matrix elements (C_{ij}) are Boolean values. If work order (i) and work order (j) involve the same or related business systems and their time difference is less than the preset time threshold, then (C_{ij}) is 1, indicating that there is a conflict between the two.

[0254] Conflict determination analysis: Analyze the Boolean conflict matrix and identify all elements marked as 1 to identify conflicting ticket pairs. For each conflicting ticket pair, evaluate the proximity of their operation times and the relevance of their business systems to comprehensively determine the conflict risk.

[0255] (4) Generation of risk warning report.

[0256] Integrate the results of time series analysis and multi-work order conflict detection to form a comprehensive assessment of the current work order status.

[0257] High-risk ticket distribution: Lists all tickets identified as high-risk, including their business systems, operation types, and estimated execution time.

[0258] Risks during specific time periods: Highlights ticket risks during peak business periods or periods of abnormal system load, helping operations teams adjust resource allocation and priorities.

[0259] Multi-work order risk response recommendations: For combinations of work orders with overlapping risks, the report provides detailed response strategy recommendations, such as adjusting the execution sequence, adding approval processes, and optimizing execution time windows.

[0260] The above analysis results are output in the form of structured reports for the operation and maintenance team to review and take action. The reports can be real-time or generated periodically, depending on the enterprise's operation and maintenance needs and workflow.

[0261] The work order risk assessment method proposed in this application has the following advantages and significant effects:

[0262] 1. High level of intelligence, enabling accurate risk identification and grading. In the embodiments of this application, the LLM model, after fine-tuning, can efficiently parse the content of work orders and automatically identify key information in work orders. Combined with the detailed analysis of specialized small models, this improves the ability to accurately judge work order risks. Traditional work order processing methods often rely on manual review or simple rule-based judgments, making it difficult to efficiently and accurately identify and quantify work order risks. The multi-factor quantitative analysis proposed in this application can perform a multi-dimensional hierarchical assessment of risks, making risk grading more accurate and significantly improving the intelligent level of risk management.

[0263] 2. Flexible response strategy generation mechanism. This application introduces a dynamic response strategy generation mechanism to generate personalized optimization suggestions based on the risk characteristics of real-time work order data, thereby enhancing the flexibility and adaptability of the system. Unlike the strategy generation method based on a fixed rule base in related technologies, the embodiment of this application can automatically generate response plans for different levels of risks based on real-time work order information, risk levels, and historical change cases. By automatically adjusting change plans, optimizing scripts, and increasing approvals, the risk response strategy is made more flexible and effective, effectively reducing the complexity of manual adjustments and improving the practicality of strategy generation.

[0264] 3. Time series and multi-work order conflict detection improves system security. In the embodiments of this application, time series analysis is used to dynamically monitor the number of work orders and risk situations in different time periods, identifying risk trends within specific time periods and providing early warning of risks during peak work order periods. Compared to the static analysis methods of related technologies, this application can promptly detect high-risk situations such as a surge in work orders and frequent system changes through dynamic time series monitoring, helping the operation and maintenance team take preventive measures to ensure the continued stability of the system.

[0265] 4. This application can also use a conflict detection algorithm to identify the risk of multiple work orders overlapping in the same system. By detecting intensive work order operations in a short period of time, it automatically prompts the operation and maintenance personnel to adjust the work order operations, preventing the system from facing stability risks caused by the overlapping operations of multiple work orders, and effectively avoiding system failures that may be caused by the concentrated operations of multiple work orders in traditional technologies.

[0266] 5. Intelligent work order risk comprehensive assessment and risk warning report. This application uses a comprehensive risk assessment mechanism to not only conduct quantitative analysis of the risks of a single work order, but also conduct vertical and horizontal comparisons of the cumulative risks of multiple work orders, thereby achieving a comprehensive risk assessment at the system level. Based on this, the system generates a detailed risk warning report to help the operation and maintenance team pre-identify and control potential concentrated risks. Compared with related technologies that can only perform risk analysis on a single work order, the comprehensive assessment function in the embodiment of this application can effectively improve the comprehensiveness and accuracy of risk warnings, help the operation and maintenance team optimize the operation and maintenance strategy and improve the rationality of the change window, and reduce system risks caused by centralized operations.

[0267] 6. Improved automation and efficiency of the operation and maintenance process. Through the method of this application, the process of operation and maintenance risk identification, classification, and response strategy generation is highly automated, significantly reducing manual intervention. In traditional operation and maintenance processes, manual review and approval often incur significant labor costs and time delays. The LLM model in the embodiment of this application can automatically handle risk assessment and strategy generation, reducing human dependence and significantly improving operation and maintenance efficiency.

[0268] Figure 6This is a structural diagram of a work order risk assessment device according to an embodiment of the present application, such as Figure 6 As shown, the device includes:

[0269] An acquisition module 602 is configured to acquire work order information of a target work order, wherein the work order information includes the business system involved in the target work order and the scope of business impact of the target work order on the business process;

[0270] Scoring module 604, configured to determine the target risk rule triggered by the work order information and determine the risk score corresponding to the target risk rule, wherein the target risk rule is used to extract risk characteristics from the work order information;

[0271] The determination module 606 is configured to determine the business system, business impact scope, and risk score as impact factors, and determine the risk level of the target work order based on the impact factors.

[0272] It should be noted that Figure 6 The work order risk assessment device shown is used to perform Figure 2 The risk assessment method of the work order shown is therefore Figure 2 The explanations in the work order risk assessment method also apply to Figure 6 The work order risk assessment device shown is not described here in detail.

[0273] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the steps of the work order risk assessment method in each embodiment of the present application.

[0274] For example, the processor performs the following functions by executing program instructions stored in the memory: obtaining the work order information of the target work order, wherein the work order information includes the business system involved in the target work order and the business impact scope of the target work order on the business process; determining the target risk rule triggered by the work order information, and determining the risk score corresponding to the target risk rule, wherein the target risk rule is used to extract risk characteristics from the work order information; determining the business system, business impact scope and risk score as influencing factors, and determining the risk level of the target work order based on the influencing factors.

[0275] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the steps of the work order risk assessment method in each embodiment of the present application by running the computer program.

[0276] An embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the work order risk assessment method in each embodiment of the present application.

[0277] An embodiment of the present application further provides a computer program, which, when executed by a processor, implements the steps of the work order risk assessment method in each embodiment of the present application.

[0278] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0279] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0280] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0281] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0282] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0283] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0284] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for assessing work order risk, characterized in that: include: Acquire work order information of a target work order, wherein the work order information includes the business system involved in the target work order and the business impact scope of the target work order on the business process; Determining a target risk rule triggered by the work order information and determining a risk score corresponding to the target risk rule, wherein the target risk rule is used to extract risk features from the work order information; The business system, the business impact scope, and the risk score are determined as impact factors, and the risk level of the target work order is determined based on the impact factors.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining historical work order data within a preset time period, wherein the historical work order data includes a timestamp for each historical work order; Sorting the historical work order data according to the timestamps to obtain continuous time series data, and cutting the time series data into time windows of a preset length; In each time window, the forecast model is used to predict the number of work orders in each time window; A target time window with risk is determined based on the predicted number of work orders and the actual number of work orders within the corresponding time window.

3. The method according to claim 2, characterized in that Determining a target time window with risk based on the predicted number of work orders and the actual number of work orders within the corresponding time window includes: Determine the difference between the predicted number of work orders and the actual number of work orders within each time window; Determining an anomaly score corresponding to each time window based on the difference, wherein the anomaly score is used to quantify the degree of fluctuation of the difference between the predicted number of work orders and the actual number of work orders; The target time window corresponding to the anomaly score is determined.

4. The method according to claim 2, characterized in that The method further comprises: Extracting the first business system involved in each work order from the work order sequence data, wherein the work order sequence data is obtained by sorting the historical work order data according to the timestamp; Determine a conflicting work order in the work order sequence data that generates a conflict with the first business system, and determine a second business system that generates the conflict from a first business system set formed by the first business system; For each second business system, a conflict risk score of each conflicting work order is determined according to the time difference between the conflicting work orders, wherein the conflict risk score is used to quantitatively represent the degree of influence of the time difference on the conflict risk.

5. The method according to claim 4, characterized in that Determining conflicting work orders generated by the first business system in the work order sequence data includes: Determine a Boolean conflict matrix, wherein rows and columns in the Boolean conflict matrix are respectively used to represent different historical work orders in the work order sequence data; If the first business system corresponding to the two historical work orders involved in each element in the Boolean conflict matrix is ​​the same and the time difference corresponding to the timestamps of the two historical work orders is less than a time threshold, the position corresponding to the element is marked as a valid value, wherein the valid value is used to indicate that the two historical work orders corresponding to the element have a conflict; The work orders corresponding to all the valid values ​​in the Boolean conflict matrix are determined as the conflicting work orders.

6. The method according to claim 1, characterized in that The work order information of the target work order is extracted using a recognition model; the recognition model is trained in the following manner: Using a first recognition model to identify the location and type of work order information in the training work order data, obtaining a first recognition result; Adjusting parameters of the first recognition model based on the first recognition result and the first label of the training work order data, and obtaining a second recognition model when the first recognition result satisfies a first preset condition, wherein the first preset condition is that the accuracy of the first recognition model in recognizing the location and type of the work order information is greater than or equal to a first preset accuracy; Using the second recognition model to extract the content of the work order information of the training work order data to obtain a second recognition result; Adjust the parameters of the second recognition model based on the second recognition result and the second label of the training work order data, and obtain the recognition model when the second recognition result meets the second preset condition, wherein the second preset condition is that the accuracy of the second recognition model in extracting the content of the work order information is greater than or equal to the second preset accuracy.

7. The method according to claim 6, characterized in that A risk prediction model is used to determine the target risk rule triggered by the work order information, and to determine the risk score corresponding to the target risk rule; the risk prediction model is trained in the following manner: Acquire a risk rule data set, wherein the risk rule data set includes structured coded risk rules; Using the recognition model, matching the risk characteristics of the work order information of the training work order data with the risk rules to obtain a third recognition result; Adjusting the parameters of the recognition model based on the third recognition result and the rule labels of the training work order data, and obtaining an initial risk prediction model when the third recognition result satisfies a third preset condition, wherein the third preset condition is that the accuracy of the recognition model in identifying the risk characteristics of the work order information is greater than or equal to a third preset accuracy; Using the initial risk prediction model to predict the risk score corresponding to the training work order data to obtain a predicted risk score; The parameters of the initial risk prediction model are adjusted based on the predicted risk score and the score label of the training work order data, and the risk prediction model is obtained when the predicted risk score meets the fourth preset condition, wherein the fourth preset condition is that the accuracy of the initial risk prediction model in predicting the risk score is greater than or equal to the fourth preset accuracy.

8. The method according to claim 1, characterized in that After determining the risk level of the target work order based on the influencing factor, the method further includes: Determining a target case that matches the risk level of the target work order from a historical case library, wherein the historical case library is used to store risk assessment records of historical work orders; Determining a risk management plan corresponding to the target case, wherein the risk management plan is used to mitigate or eliminate the risk of the work order; A target risk management plan corresponding to the target work order is determined according to the risk management plan and the risk level.

9. The method according to claim 1, characterized in that Determining the business system, the business impact scope, and the risk score as impact factors, and determining the risk level of the target work order based on the impact factors, including: Determine a first characteristic value corresponding to the type of the business system and a second characteristic value corresponding to the business impact range; Obtaining weights corresponding to the first feature value, the second feature value, and the risk score, respectively, and determining a target score for the target work order based on the weights; The target score is mapped to a preset risk level classification to obtain the risk level.

10. A work order risk assessment device, characterized in that: include: An acquisition module, configured to acquire work order information of a target work order, wherein the work order information includes the business system involved in the target work order and the scope of business impact of the target work order on the business process; a scoring module, configured to determine a target risk rule triggered by the work order information and determine a risk score corresponding to the target risk rule, wherein the target risk rule is used to extract risk features from the work order information; A determination module is configured to determine the business system, the business impact scope, and the risk score as impact factors, and determine the risk level of the target work order based on the impact factors.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the work order risk assessment method described in any one of claims 1 to 9.

12. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the work order risk assessment method according to any one of claims 1 to 9 by running the computer program.

13. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the work order risk assessment method described in any one of claims 1 to 9 is implemented.

Images