Abnormality processing method and device, electronic equipment and storage medium
By recording and learning operational action data during anomaly handling, process model patches are generated, and the process model is automatically repaired and iterated. This solves the problem that the process model in the prior art requires manual intervention in abnormal situations, and improves the level of intelligence and operational accuracy.
Patent Information
- Application Number
- CN202411108188.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-22
AI Technical Summary
Existing process models require manual intervention and reconstruction in abnormal situations, exhibiting low levels of intelligence and an inability to iterate and repair autonomously.
By recording operational data during anomaly handling, deep neural networks are used to train and generate process model patches, automatically repairing and iterating the process model to achieve autonomous handling of anomalies.
It has improved the intelligence level of the production control process, realized the autonomous repair and iteration of the process model, reduced manual intervention, and improved the accuracy and cleanliness of operation.
Smart Images

Figure CN119002426A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of intelligent production and artificial intelligence technology, and more specifically, to an exception handling method, device, electronic device, storage medium and program product. Background Art
[0002] With the development of intelligent production, it has become a development trend to build process models for various production processes to achieve various controls on actual production. The process models in related technologies are generally designed and generated according to established models. When abnormal problems occur, manual intervention is required and the process model needs to be rebuilt, which has a low degree of intelligence. Summary of the invention
[0003] In view of this, the present disclosure provides an exception handling method, device, electronic device, storage medium and program product.
[0004] One aspect of the present disclosure provides an exception handling method, comprising:
[0005] In response to detecting that an abnormal problem occurs in the target process model during the process of controlling the target process operation, when it is determined that the abnormal database does not contain reference data for handling the abnormal problem, a processing process for handling the abnormal problem is recorded to obtain action data, wherein the action data includes at least one operation action, and the operation action is an operation for handling the abnormal problem;
[0006] Using the motion data to train the initial patch model to generate a process model patch; and
[0007] The target process model is repaired by using the process model patch to obtain the repaired target process model.
[0008] Another aspect of the present disclosure provides an exception handling device, comprising:
[0009] a recording module, configured to, in response to detecting that an abnormal problem occurs in the target process model during the process of controlling the target process operation, record a processing process for processing the abnormal problem when it is determined that the abnormal database does not contain reference data for processing the abnormal problem, and obtain action data, wherein the action data includes at least one operation action, and the operation action is an operation for processing the abnormal problem;
[0010] A first generation module is used to train the initial patch model using the action data to generate a process model patch; and
[0011] The repair module is used to repair the target process model by using the process model patch to obtain the repaired target process model.
[0012] Another aspect of the present disclosure provides an electronic device, comprising:
[0013] one or more processors;
[0014] a memory for storing one or more programs,
[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.
[0016] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.
[0017] Another aspect of the present disclosure provides a computer program product, which includes computer executable instructions, and when the instructions are executed, are used to implement the method as described above.
[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0020] Figure 1 is a flowchart of an exception handling method according to an embodiment of the present disclosure;
[0021] Figure 2 is a flow chart of a method for repairing a process model according to an embodiment of the present disclosure;
[0022] Figure 3 is a flowchart of a method for handling an exception when a branch operation occurs according to an embodiment of the present disclosure;
[0023] Figure 4 is a flow chart of a method for repairing a process model according to another embodiment of the present disclosure;
[0024] Figure 5 is a flowchart of an exception handling method for an erroneous operation according to an embodiment of the present disclosure;
[0025] Figure 6 is a flow chart of a method for handling an exception using an action model according to an embodiment of the present disclosure;
[0026] Figure 7 is a flowchart of an exception handling method according to another embodiment of the present disclosure;
[0027] Figure 8 is a schematic diagram of an exception handling method according to an embodiment of the present disclosure;
[0028] Fig. 9 is a block diagram of an exception handling device according to an embodiment of the present disclosure; and
[0029] Fig.10 is a block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of them. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present disclosure. It should be noted that throughout the drawings, the same elements are represented by the same or similar figure marks. In the following description, some specific embodiments are only used for descriptive purposes and should not be understood as any limitation to the present disclosure, but are only examples of the embodiments of the present disclosure. Conventional structures or configurations will be omitted when they may cause confusion in the understanding of the present disclosure. It should be noted that the shapes and sizes of the components in the figures do not reflect the actual size and proportion, but only illustrate the contents of the embodiments of the present disclosure.
[0031] Unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by those skilled in the art. The words "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components.
[0032] Remote Control System (RCM) is a high-performance, high-stability and reliable system with hardware and software architecture. RCM centralized control system can realize synchronous alarm light status monitoring and machine interface control operation functions for multiple devices through a set of keyboard, mouse and a monitor. RCM centralized control system has functions such as remote operation, alarm processing, fault tracing, equipment status monitoring and equipment data copying, remote technical support, and artificial intelligence operation.
[0033] In the production process of display panels, semiconductors, wafers, etc., the cleanliness requirements are very high, and the biggest impact on cleanliness is often the participation of personnel. The lower the human participation, the higher the cleanliness of the workshop. Therefore, more and more manufacturing industries are building unmanned factories.
[0034] Manual operation is to intelligently model some common and commonly used process flows and build a set of mature process models. Operators can remotely control the equipment in the production line through RCM. Through a set of process models, manual operation can be realized, and one person and one computer can operate multiple production lines and multiple processes. This can minimize the participation of manual labor in workshop production, improve cleanliness, reduce labor input, and increase the accuracy of operation.
[0035] However, the process model modeling system in the related art generally adopts a point-to-point modeling method, which is to establish a process model according to a predetermined design model through manual operation. The process model established in this way has the following technical problems: the process model is inaccurate, and when abnormal problems occur during the model operation, manual intervention is required and the process model needs to be rebuilt. For example, if an erroneous operation occurs during the implementation of the process model, manual intervention is required to handle the problem and rebuild the model, and the iterative model cannot be modified independently; for example, when a branch operation or abnormal alarm occurs during the operation of the process model, manual guidance or abnormal processing is required, resulting in low intelligence and simplification of the process model.
[0036] In response to the above technical problems, the present disclosure provides an exception handling method, including: in response to detecting that an abnormal problem occurs in the target process model during the process of controlling the target process operation, when determining that the abnormal database does not contain reference data for handling the abnormal problem, recording the processing process for handling the abnormal problem to obtain action data, wherein the action data includes at least one operation action, and the operation action is an operation for handling the abnormal problem; using the action data to train the initial patch model to generate a process model patch; and using the process model patch to repair the target process model to obtain the repaired target process model. By recording the processing process for handling the abnormal problem, using the processing process to train the patch model to learn the action to generate a process model patch, and then using the process model patch to repair the target process model, automatic modification and repair of the model is achieved, the process model is intelligently iterated, and the immature process model is continuously updated. As the production learning proceeds, a mature, stable intelligent model that can autonomously handle abnormal problems is finally iterated to generate, thereby building a set of mature process model libraries.
[0037] Figure 1 is a flowchart of an exception handling method according to an embodiment of the present disclosure.
[0038] like Figure 1 As shown, the method includes operations S110 to S130.
[0039] In operation S110, in response to detecting that an abnormal problem occurs in the target process model during the control of the target process operation, when it is determined that the abnormal database does not contain reference data for handling the abnormal problem, a processing process for handling the abnormal problem is recorded to obtain action data, wherein the action data includes at least one operation action, and the operation action is an operation for handling the abnormal problem.
[0040] According to an embodiment of the present disclosure, the target process model can be a process model selected by the remote control system from a model library for implementing the target process operation. After the target process model is selected, it is imported into the production system to replace manual target process operation.
[0041] According to an embodiment of the present disclosure, the target process model may be a work order issuing process model, a material handling process model, a laser process model, a detection process model, etc., wherein the material handling process model may include a feeding model, the laser process model may include a cutting model, a line drawing model, a welding model, a repair model, etc., and the detection process model may include an image acquisition model, an image analysis model, etc.
[0042] According to an embodiment of the present disclosure, the exception database may be a database containing exception processing data of all abnormal problems that occurred during the manual operation of the process model. For example, process model 1 had abnormal problems a1 and b1 during the manual operation; process model 2 had abnormal problems a2 and b2 during the manual operation, then the exception database may include abnormal processing data for processing abnormal problem a1, abnormal processing data for processing abnormal problem b1, abnormal processing data for processing abnormal problem a2, and abnormal processing data for processing abnormal problem b2.
[0043] According to an embodiment of the present disclosure, the processing process for handling abnormal problems may include operating actions of an operator in the process of handling abnormal problems. The operating actions may include mouse events, keyboard events, etc. For example, the operating actions may be mouse click positions, keyboard key positions, etc.
[0044] In some embodiments, the motion data can be generated by recording the mouse events and keyboard events. In other embodiments, the motion data can be generated by recording the screen of the operation interface and based on the screen recording video.
[0045] According to an embodiment of the present disclosure, the abnormal problem may include the occurrence of an erroneous operation or the occurrence of a branch operation.
[0046] According to an embodiment of the present disclosure, an erroneous operation may include the target process model implementing an erroneous operation in the process of controlling the target process operation. For example, if the feeding model does not add material when it is required to add material during the target process operation, it is considered that an erroneous operation has occurred; for another example, if the amount of material added by the feeding model is inaccurate, it is considered that an erroneous operation has occurred.
[0047] According to the embodiments of the present disclosure, during the production process, products are processed sequentially according to a pre-set process flow, wherein branch processes often exist in the process flow. When a target process model encounters a branch process while controlling the target process operation, situations that cannot be handled may easily occur, requiring manual guidance of the operation, resulting in abnormal problems in branch operations.
[0048] In some embodiments, for example, two branch processes are separated at process site A of the process flow, such as branch process 1 and branch process 2. At this time, when the product flows to process site A, it is necessary to determine whether to enter branch process 1 or branch process 2, that is, a branch operation occurs. Specifically, for example, branch process 1 is used for cutting, and branch process 2 is used for welding. If the product flows to process site A and a short circuit occurs in the product, it is necessary to enter branch process 1 for cutting operation; if the product flows to process site A and the product is in an open state, it is necessary to enter branch process 2 for welding operation.
[0049] In operation S120 , the initial patch model is trained using the motion data to generate a process model patch.
[0050] According to an embodiment of the present disclosure, the initial patch model may be a deep neural network model, and the initial patch model may be trained using action data. Generating a process model patch may include: importing the action data into the deep neural network model, and using a deep learning method to learn the operating actions in the action data, thereby generating a process model patch.
[0051] In operation S130 , the target process model is repaired by using the process model patch to obtain a repaired target process model.
[0052] According to an embodiment of the present disclosure, repairing a target process model using a process model patch may include inserting the process model patch into a location where an abnormal problem occurs in the target process model; or replacing the location where an abnormal problem occurs in the target process model using the process model patch.
[0053] According to an embodiment of the present disclosure, when the abnormal problem is an erroneous operation, repairing the target process model using a process model patch may be to use the process model patch to replace the location where the abnormal problem occurs in the target process model, thereby avoiding the erroneous operation of the repaired target process model again.
[0054] According to an embodiment of the present disclosure, when the abnormal problem is the occurrence of a branch operation, repairing the target process model using a process model patch may be to insert the process model patch into the location where the abnormal problem occurs in the target process model, thereby enabling the repaired target process model to autonomously handle the branch operation when encountering a branch operation.
[0055] According to the embodiments of the present disclosure, in response to detecting that an abnormal problem occurs in the target process model during the control of the target process operation, when it is determined that the abnormal database does not contain reference data for handling the abnormal problem, the processing process for handling the abnormal problem is recorded to obtain action data; then the initial patch model is trained using the action data to generate a process model patch; thereafter, the target process model is repaired using the process model patch to obtain a technical solution for the repaired target process model, thereby realizing autonomous modification and repair of the model and iteratively generating a new process model, thereby obtaining a process model that can autonomously handle abnormal problems and improving the intelligence level of the production control process.
[0056] According to an embodiment of the present disclosure, the method further includes: in response to detecting that an abnormal problem occurs in the target process model during the control of the target process operation, stopping the target process model from controlling the target process operation and sending abnormal alarm information.
[0057] According to an embodiment of the present disclosure, the abnormal alarm information may include the location information of the abnormal problem and the process model information where the abnormal problem occurs, so that the operator can determine the abnormal problem according to the abnormal alarm information and then handle the abnormal problem.
[0058] According to an embodiment of the present disclosure, recording a processing process for handling an abnormal problem and obtaining action data includes: starting a screen recording function of an operation interface corresponding to a target process operation, wherein the screen recording function is used to record at least one operation action performed on the operation interface; in response to an operation instruction for completing the handling of the abnormal problem, stopping the screen recording function and generating action data.
[0059] According to the embodiments of the present disclosure, each process operation has a corresponding operation interface. When an abnormal problem occurs during the process of the target process model controlling the target process operation, the screen recording function of the operation interface corresponding to the target process operation is triggered to record the operation process of the operator on the operation interface. After the operator completes the processing of the abnormal problem, the screen recording function is automatically stopped, and the screen recording data from starting the screen recording function to stopping the screen recording function is used as the action data.
[0060] According to an embodiment of the present disclosure, the action data includes screen recording data, which may be a screen recording video that utilizes a screen recording function to record the operation process on an operation interface. The action data is used to train the initial patch model, and generating a process model patch includes: segmenting the action data to obtain at least one action image, wherein the action image includes the operation position of the operation action; iteratively training the initial patch model using at least one action image so that the initial patch model learns the operation position of the operation action; and determining that the model obtained when the initial patch model reaches an iteration stop condition is the process model patch.
[0061] According to an embodiment of the present disclosure, segmenting the action data to obtain at least one action image may include segmenting the action data using a preset segmentation algorithm to obtain at least one action image. The segmentation algorithm is not limited as long as the action data can be segmented into multiple images each containing an operation action.
[0062] According to an embodiment of the present disclosure, the operating action in the operating position of the operating action may include clicking a mouse or pressing a keyboard, and the operating position may include a position where the mouse is clicked or a position where a keyboard key is pressed.
[0063] According to an embodiment of the present disclosure, segmenting the action data to obtain at least one action image includes: segmenting the action data to obtain segmented images, and selecting an image containing the operation position of the operation action from the segmented images as the action image. For example, the action data is segmented to obtain 100 frames of images. Since the screen recording data is coherent, there are images in the 100 frames that do not contain the operation action, that is, images that the operator has not operated. Therefore, the image containing the operation action is selected from the 100 frames as the action image.
[0064] According to an embodiment of the present disclosure, since the initial patch model needs to learn the operation position of the operation action, selecting images containing the operation action as training samples helps to improve the learning efficiency of the initial patch model and improve the accuracy of the model.
[0065] Figure 2 is a flow chart of a method for repairing a process model according to an embodiment of the present disclosure.
[0066] According to an embodiment of the present disclosure, when the abnormal problem is a branch operation, the repair method for the process model is as follows: Figure 2 As shown, the repairing method of the process model of this embodiment includes operations S210 to S220.
[0067] In operation S210 , a position to be repaired of a target process model is determined according to the position information.
[0068] In operation S220 , the process model patch is inserted into the to-be-repaired position of the target process model to obtain a repaired target process model.
[0069] According to an embodiment of the present disclosure, the location where the branch operation occurs can be determined based on the position information, and the location can be used as the location to be repaired. Then, by inserting the process model patch into the location to be repaired, the repaired target process model can perform a branch operation according to the operation learned by the process model patch when encountering a branch operation next time, thereby obtaining an intelligent model that can autonomously handle abnormal problems.
[0070] Figure 3 The flowchart of the method for handling an exception when a branch operation occurs according to an embodiment of the present disclosure is shown.
[0071] like Figure 3 As shown, the exception handling method of this embodiment includes operations S301 to S310.
[0072] In operation S301, the remote control system imports a target process model from a model library and performs manual operation.
[0073] In operation S302, it is determined whether the target process model has a branch operation to be confirmed during the manual operation process. If so, operation S303 is executed, and if not, operation S310 is executed.
[0074] In operation S303, the remote control system stops the operation of the target process model and sends abnormal alarm information, wherein the abnormal alarm information includes abnormal position information.
[0075] In operation S304, a screen recording function of the operation interface is started to record the operation actions of the operator on the operation interface.
[0076] In operation S305 , in response to the operator completing the operation on the operation interface, the screen recording function is stopped and screen recording data is generated.
[0077] In operation S306, the screen recording data is segmented to obtain at least one action image, wherein the action image includes an operation position of the operation action.
[0078] In operation S307, the initial patch model is iteratively trained using at least one action image, so that the initial patch model learns the operation position of the operation action and generates a process model patch.
[0079] In operation S308, the process model patch is inserted into the abnormal position of the target process model to obtain a repaired target process model, and the repaired target process model is used to continue the manual operation.
[0080] In operation S309, it is determined whether the repaired target process model has a branch operation to be confirmed during the manual operation process. If so, operation S303 is executed, and if not, operation S310 is executed.
[0081] In operation S310, a stable process model is output and stored in a model library.
[0082] According to an embodiment of the present disclosure, when a branch operation to be confirmed appears in the target process model, the operation of the operator on the operation interface is recorded and the initial patch model is trained using the operation process to learn the action and generate a process model patch, and then the target process model is repaired to generate a new model, thereby realizing automatic modification and repair of the model and iterative generation of a new model, thereby obtaining a model that can handle problems autonomously.
[0083] Figure 4 is a flow chart of a method for repairing a process model according to another embodiment of the present disclosure.
[0084] According to an embodiment of the present disclosure, when the abnormal problem is an erroneous operation, the repair method for the process model is as follows: Figure 4 As shown, the repairing method of the process model of this embodiment includes operations S410 to S420.
[0085] In operation S410 , a position to be repaired of a target process model is determined according to the position information.
[0086] In operation S420, the model data of the to-be-repaired position of the target process model is replaced by the process model patch to obtain a repaired target process model.
[0087] According to the embodiments of the present disclosure, the location where the erroneous operation occurred can be determined based on the location information, and the location is used as the location to be repaired. Then, the model parameters of the location to be repaired are replaced with a process model patch, thereby realizing automatic model modification and repair.
[0088] Figure 5 The figure is a flowchart of a method for handling an exception when an erroneous operation occurs according to an embodiment of the present disclosure.
[0089] like Figure 5 As shown, the exception handling method of this embodiment includes operations S501 to S510.
[0090] In operation S501, the remote control system imports a target process model from a model library and performs manual operation.
[0091] In operation S502, it is determined whether an erroneous operation occurs in the target process model during the manual operation. If so, operation S503 is executed, and if not, operation S510 is executed.
[0092] In operation S503, the remote control system stops the operation of the target process model and sends abnormal alarm information, wherein the abnormal alarm information includes abnormal position information.
[0093] In operation S504, a screen recording function of the operation interface is started to record the operation actions of the operator on the operation interface.
[0094] In operation S505, in response to the operator completing the operation on the operation interface, the screen recording function is stopped and screen recording data is generated.
[0095] In operation S506, the screen recording data is segmented to obtain at least one action image, wherein the action image includes an operation position of the operation action.
[0096] In operation S507, the initial patch model is iteratively trained using at least one action image, so that the initial patch model learns the operation position of the operation action and generates a process model patch.
[0097] In operation S508, the process model patch is used to replace the parameters of the abnormal position of the target process model to obtain a repaired target process model, and the repaired target process model is used to continue the manual operation.
[0098] In operation S509, it is determined whether the repaired target process model has any erroneous operation during the manual operation process. If so, operation S503 is executed, and if not, operation S510 is executed.
[0099] In operation S510, a stable process model is output and stored in a model library.
[0100] According to an embodiment of the present disclosure, when an erroneous operation occurs in the target process model, the operation of the operator on the operation interface is recorded and the initial patch model is trained using the operation process to learn the action to generate a process model patch, and then the erroneous position parameters of the target process model are replaced to generate a new model, thereby realizing automatic modification and repair of the model and iterative generation of a new model, thereby obtaining a model that can handle problems autonomously.
[0101] According to an embodiment of the present disclosure, the above method also includes: generating exception handling reference data corresponding to the exception alarm information based on the exception alarm information and action data; sending the exception handling reference data to the exception database; and generating an action model using the exception handling reference data, wherein the action model is used to handle exception problems based on the exception handling reference data.
[0102] According to an embodiment of the present disclosure, the method further includes: when it is determined that the exception database contains reference data for handling exception problems, updating the action model using the exception handling reference data to obtain an updated action model.
[0103] According to an embodiment of the present disclosure, generating an action model using exception handling reference data may include training an initial action model using exception handling parameter data to obtain an action model. As the exception handling parameter data in the exception database continues to increase, the action model is continuously iterated and updated using the increased data to obtain an updated action model.
[0104] Figure 6 It is a flowchart of a method for handling exceptions using an action model according to an embodiment of the present disclosure.
[0105] According to an embodiment of the present disclosure, in response to detecting that an abnormal problem occurs in the target process model during the process of controlling the target process operation, when it is determined that the abnormal database contains reference data for handling the abnormal problem, the following method can be used: Figure 6 The exception handling method shown.
[0106] like Figure 6 As shown, the exception handling method of this embodiment may include operations S610 to S660.
[0107] In operation S610, reference data for handling abnormal problems contained in an abnormal database is acquired to obtain target reference data.
[0108] In operation S620, the action model is called, and the abnormal problem is automatically processed using the action model according to the target reference data.
[0109] In operation S630, the number of times the abnormal problem occurs is counted.
[0110] In operation S640, it is determined whether the number of occurrences of the abnormal problem is greater than or equal to a preset threshold; if not, operation S650 is performed; if so, operation S660 is performed.
[0111] In operation S650, abnormality handling reference data corresponding to the abnormal problem is generated and the action model is updated using the abnormality handling reference data to obtain an updated action model.
[0112] In operation S660, the updated action model is added to the model library as a new process model for controlling at least one process operation. According to an embodiment of the present disclosure, counting the number of occurrences of abnormal problems can be counting the number of occurrences of the current abnormal problem. When the number of occurrences of the current abnormal problem is greater than or equal to a preset threshold, it indicates that the number of occurrences of the abnormal problem is large. At this time, the action model can be added to the model library as a new process model. In addition, when new abnormal processing reference data is added to the abnormal database, the action model can be autonomously updated using the newly added reference data. After repeated learning, it can be autonomously iterated into a process model with autonomous abnormal problem processing.
[0113] According to an embodiment of the present disclosure, before the action model is added to the model library, after an abnormal problem occurs, the action model can be called to solve the abnormal problem. At this time, since the number of abnormal problems does not occur often enough and the number of iterations of the action model is limited, it does not need to be added to the model library. However, when the number of abnormal problems occurs is large, it indicates that the action model has generated a model with the ability to autonomously handle abnormal problems through multiple iterations. At this time, the action model after multiple iterations can be added to the model library as a new process model. After the action model is added to the model library, it and the target process model both belong to one of the process models in the model library. In the subsequent production process, it can be coordinated with the target process model for manual operation. Since the action model can solve the abnormal problem, the alarm information of the abnormal problem can be avoided.
[0114] According to an embodiment of the present disclosure, the above method further includes: replacing the target process model in the model library with the repaired target process model, wherein the target process model is one of the process models in the model library.
[0115] According to the embodiments of the present disclosure, by continuously iterating the model, the model in the model library is replaced with the repaired target process model, and unqualified models in the model library are eliminated, and finally a model library that can autonomously handle branch actions and alarms is generated.
[0116] Figure 7 is a flowchart of an exception handling method according to another embodiment of the present disclosure.
[0117] like Figure 7 As shown, the exception handling method of this embodiment includes operations S710 to S714.
[0118] In operation S701, the remote control system imports a target process model from a model library and performs manual operation.
[0119] In operation S702, it is determined whether an abnormal problem occurs in the target process model during the manual operation process. If so, operation S703 is performed, and if not, operation S714 is performed.
[0120] In operation S703, it is determined whether the abnormal problem occurs in the abnormal database; if so, operation S712 is executed; if not, operation S704 is executed.
[0121] In operation S704, the remote control system stops the operation of the target process model and sends abnormal alarm information.
[0122] In operation S705, a screen recording function of the operation interface is started to record the operation actions of the operator on the operation interface.
[0123] In operation S706 , in response to the operator completing the operation on the operation interface, the screen recording function is stopped, action data is generated, and then operations S706 to S708 and operations S709 to S711 are respectively executed.
[0124] In operation S707, abnormality handling reference data is generated according to the action data and the abnormality alarm information.
[0125] In operation S708 , the exception handling reference data is sent to an exception database, and an action model is generated according to the exception handling reference data.
[0126] In operation S709, segmentation processing is performed on the action data to obtain at least one action image, wherein the action image includes an operation position of the operation action.
[0127] In operation S710, an initial patch model is iteratively trained using at least one action image to generate a process model patch.
[0128] In operation S711, the target process model is repaired by using the process model patch to obtain the repaired target process model, and the repaired target process model is used to continue the manual operation, and then operation S702 is performed.
[0129] In operation S712, the abnormal problem is processed using the action model.
[0130] In operation S713, the number of occurrences of abnormal problems is counted, and when the number of occurrences of abnormal problems is greater than or equal to a preset threshold, operation S714 is performed. In operation S714, a stable process model is output and stored in a model library.
[0131] According to the embodiments of the present disclosure, in addition to using the action data to train the initial patch model to modify the target process model, the action data is used to generate exception handling reference data and stored in the exception database, so that the action model is generated according to the exception handling reference data to achieve autonomous handling of exception problems. At the same time, the action model can be autonomously updated according to the newly added data, and after repeated learning, it can autonomously iterate and generate a process model with autonomous awareness.
[0132] Figure 8 It is a schematic diagram of an exception handling method according to an embodiment of the present disclosure.
[0133] like Figure 8 As shown, the model library 810 includes multiple process models, such as process model 1, process model 2, process model 3, process model 4, ..., process model n; the remote control system selects the target process model from the model library 810 according to the target process operation, and implements manual operation 820 for the target process model; when an abnormal alarm occurs during the implementation of manual operation of the target process model, the manual operation is interrupted and the screen recording function is started 830; after the operator performs an abnormal handling operation on the operation interface, the screen recording function is stopped and action data 840 is generated, which is a screen recording video for the operation interface; thereafter, the operation actions in the action data are learned by using a deep learning method to repair the target process model 850, and finally the repaired target process model is stored in the model library 810.
[0134] According to an embodiment of the present disclosure, there are several pre-generated process models in the model library. When an abnormal problem occurs during the implementation of a process model, the operator's processing of the human guidance or abnormal handling operation of the process model is recorded to generate video data. Then, the operation action is learned through a deep learning method and filled in the process model, and an empirical model capable of handling abnormal and branch operations is iteratively generated. This method will repeat the above operations on the iteratively generated model, continuously iterate the model, and eliminate unqualified models, and finally generate a model library that can autonomously handle branch actions and alarm processing.
[0135] Fig. 9 is a block diagram of an exception handling apparatus according to an embodiment of the present disclosure.
[0136] like Fig. 9 As shown, the exception handling device of this embodiment includes: a recording module 910 , a first generating module 920 and a repairing module 930 .
[0137] The recording module 910 is used for, in response to detecting that an abnormal problem occurs in the target process model during the process of controlling the target process operation, recording a processing process for processing the abnormal problem when it is determined that the abnormal database does not contain reference data for processing the abnormal problem, and obtaining action data, wherein the action data includes at least one operation action, and the operation action is an operation for processing the abnormal problem.
[0138] The first generating module 920 is used to train the initial patch model using the action data to generate a process model patch.
[0139] The repair module 930 is used to repair the target process model by using the process model patch to obtain the repaired target process model.
[0140] According to an embodiment of the present disclosure, the above-mentioned device also includes: a first sending module.
[0141] The first sending module is used to stop the target process model from controlling the target process operation and send abnormal alarm information in response to detecting that an abnormal problem occurs in the target process model during the process of controlling the target process operation.
[0142] According to an embodiment of the present disclosure, the abnormal alarm information includes location information of the abnormal problem.
[0143] According to an embodiment of the present disclosure, the repair module includes: a first determination submodule, an insertion submodule and a replacement submodule.
[0144] The first determination submodule is used to determine the to-be-repaired position of the target process model according to the position information.
[0145] The inserting submodule is used to insert the process model patch into the position to be repaired of the target process model to obtain the repaired target process model.
[0146] The replacement submodule is used to replace the model data of the to-be-repaired position of the target process model with the process model patch to obtain the repaired target process model.
[0147] According to an embodiment of the present disclosure, the abnormal problem includes the occurrence of an erroneous operation or the occurrence of a branch operation.
[0148] According to an embodiment of the present disclosure, the action data includes screen recording data, and the screen recording data includes a screen recording video that records the operation process on the operation interface using a screen recording function.
[0149] According to an embodiment of the present disclosure, the first generation module includes: a segmentation submodule, a training submodule and a second determination submodule.
[0150] The segmentation submodule is used to perform segmentation processing on the action data to obtain at least one action image, wherein the action image includes the operation position of the operation action.
[0151] The training submodule is used to iteratively train the initial patch model using at least one action image so that the initial patch model learns the operation position of the operation action.
[0152] The second determination submodule is used to determine that the model obtained when the initial patch model reaches the iteration stop condition is the process model patch.
[0153] According to an embodiment of the present disclosure, the above-mentioned device further includes: a second generating module, a second sending module and a third generating module.
[0154] The second generating module is used to generate exception processing reference data corresponding to the exception alarm information according to the exception alarm information and the action data.
[0155] The second sending module is used to send the exception processing reference data to the exception database.
[0156] The third generating module is used to generate an action model using the exception handling reference data, wherein the action model is used to handle the exception problem according to the exception handling reference data.
[0157] According to an embodiment of the present disclosure, the above-mentioned device also includes: an acquisition module and a calling module.
[0158] The acquisition module is used to acquire the reference data to obtain the target reference data when it is determined that the abnormal database contains reference data for handling abnormal problems.
[0159] The calling module is used to call the action model and automatically handle abnormal problems using the action model according to the target reference data.
[0160] According to an embodiment of the present disclosure, the above-mentioned device also includes: an update module.
[0161] The updating module is used to update the action model using the exception processing reference data to obtain the updated action model when it is determined that the exception database contains reference data for processing the exception problem.
[0162] According to an embodiment of the present disclosure, the above-mentioned device also includes: a statistical module and an adding module.
[0163] The statistics module is used to count the number of abnormal problems that occur.
[0164] The adding module is used to add the updated action model to the model library as a new process model for controlling at least one process operation when the number of abnormal problems occurring is greater than or equal to a preset threshold.
[0165] According to an embodiment of the present disclosure, the above-mentioned device also includes: a replacement module.
[0166] The replacement module is used to replace the target process model in the model library with the repaired target process model, wherein the target process model is one of the process models in the model library.
[0167] According to an embodiment of the present disclosure, the recording module includes: a start submodule and a stop and generate submodule.
[0168] The startup submodule is used to start the screen recording function of the operation interface corresponding to the target process operation, wherein the screen recording function is used to record at least one operation action performed on the operation interface.
[0169] The stop and generate submodule is used to respond to the operation instruction completed for the abnormal problem processing, stop the screen recording function, and generate action data.
[0170] According to the modules and submodules of the embodiments of the present disclosure, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules and submodules of the embodiments of the present disclosure, any one or more thereof can be split into multiple modules for implementation. According to the modules and submodules of the embodiments of the present disclosure, any one or more thereof can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, according to the modules and submodules of the embodiments of the present disclosure, one or more thereof can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.
[0171] For example, any multiple of the recording module 910, the first generation module 920, and the repair module 930 can be combined in one module / unit / subunit for implementation, or any one of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more of these modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to an embodiment of the present disclosure, at least one of the recording module 910, the first generation module 920, and the repair module 930 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, at least one of the recording module 910, the first generating module 920 and the repairing module 930 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.
[0172] It should be noted that the exception handling device part in the embodiment of the present disclosure corresponds to the exception handling method part in the embodiment of the present disclosure. The description of the exception handling device part specifically refers to the exception handling method part, which will not be repeated here.
[0173] Fig.10 is a block diagram of an electronic device suitable for implementing the method described above according to an embodiment of the present disclosure. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0174] like Fig.10 As shown, the electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0175] In RAM 1003, various programs and data required for the operation of electronic device 1000 are stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1002 and / or RAM 1003. It should be noted that the program can also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.
[0176] According to an embodiment of the present disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may further include one or more of the following components connected to the input / output (I / O) interface 1005: an input portion 1006 including a keyboard, a mouse, etc.; an output portion 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk, etc.; and a communication portion 1009 including a network interface card such as a LAN card, a modem, etc. The communication portion 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage portion 1008 as needed.
[0177] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.
[0178] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0179] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.
[0180] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 1002 and / or the RAM 1003 described above and / or one or more memories other than the ROM 1002 and the RAM 1003 .
[0181] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the exception handling method provided by the embodiment of the present disclosure.
[0182] When the computer program is executed by the processor 1001, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.
[0183] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0184] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0185] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.
[0186] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. An exception handling method, include: In response to detecting that an abnormal problem occurs in the target process model during the process of controlling the target process operation, when it is determined that the abnormal database does not contain reference data for processing the abnormal problem, a processing process for processing the abnormal problem is recorded to obtain action data, wherein the action data includes at least one operation action, and the operation action is an operation for processing the abnormal problem; Using the motion data to train the initial patch model to generate a process model patch; and The target process model is repaired by using the process model patch to obtain a repaired target process model.
2. The method according to claim 1, further comprising: include: In response to detecting that an abnormal problem occurs in the target process model during the process of controlling the target process operation, the control of the target process operation by the target process model is stopped, and abnormal alarm information is sent.
3. The method according to claim 2, in, The abnormal alarm information includes the location information of the abnormal problem, and the process model patch is used to repair the target process model to obtain the repaired target process model, including: Determine the position to be repaired of the target process model according to the position information; inserting the process model patch into the to-be-repaired position of the target process model to obtain the repaired target process model; or The process model patch is used to replace the model data of the to-be-repaired position of the target process model to obtain the repaired target process model.
4. The method according to claim 1, in, The abnormal problem includes an erroneous operation or a branch operation.
5. The method according to claim 1, in, The action data includes screen recording data, wherein the screen recording data includes a screen recording video of an operation process on an operation interface using a screen recording function; and the using the action data to train the initial patch model to generate a process model patch includes: Segmenting the action data to obtain at least one action image, wherein the action image includes an operation position of the operation action; Iteratively training the initial patch model using the at least one action image so that the initial patch model learns the operation position of the operation action; It is determined that the model obtained when the initial patch model reaches the iteration stop condition is the process model patch.
6. The method according to claim 2, further comprising: include: Generate exception handling reference data corresponding to the exception alarm information according to the exception alarm information and the action data; Sending the exception handling reference data to the exception database; An action model is generated using the exception handling reference data, wherein the action model is used to handle the exception problem according to the exception handling reference data.
7. The method according to claim 6, further comprising: include: In the case where it is determined that the abnormal database contains reference data for processing the abnormal problem, acquiring the reference data to obtain target reference data; The action model is called, and the abnormal problem is automatically processed using the action model according to the target reference data.
8. The method according to claim 6, further comprising: include: In the case where it is determined that the abnormality database contains reference data for processing the abnormality problem, the action model is updated using the abnormality processing reference data to obtain an updated action model.
9. The method according to claim 8, further comprising: include: Count the number of times the abnormal problem occurs; When the number of occurrences of the abnormal problem is greater than or equal to a preset threshold; The updated motion model is added to the model library as a new process model for controlling at least one process operation.
10. The method according to claim 1, further comprising: include: The target process model in the model library is replaced by the repaired target process model, wherein the target process model is one of the process models in the model library.
11. The method according to claim 1, in, The record is used to process the abnormal problem, and the action data obtained includes: Starting a screen recording function of an operation interface corresponding to the target process operation, wherein the screen recording function is used to record at least one operation action performed on the operation interface; In response to the operation instruction for completing the processing of the abnormal problem, the screen recording function is stopped and the action data is generated.
12. An exception handling device, include: a recording module, configured to, in response to detecting that an abnormal problem occurs in the target process model during the process of controlling the target process operation, record a processing process for processing the abnormal problem when it is determined that the abnormal database does not contain reference data for processing the abnormal problem, and obtain action data, wherein the action data includes at least one operation action, and the operation action is an operation for processing the abnormal problem; A first generating module, configured to train an initial patch model using the action data to generate a process model patch; and The repair module is used to repair the target process model by using the process model patch to obtain the repaired target process model.
13. An electronic device, include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 11.
14. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 11.
15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.
