An intelligent fault recovery method for industrial manufacturing equipment

Abstract

The present invention discloses an intelligent fault recovery method for industrial manufacturing equipment, including: deploying an edge device outside the manufacturing equipment, the edge device is simulated into a HID mouse and keyboard through a UDC driver, and the manufacturing equipment is operated through the simulated HID mouse and keyboard; the edge device collects relevant data of the manufacturing equipment in real time, and analyzes the acquired data through a built-in AI algorithm to determine the corresponding fault information, and has the ability of intelligent analysis and identification through learning and training of historical data; the edge device searches for the corresponding simulation technicians through image recognition based on the analysis results of the AI ​​algorithm, and the technicians directly perform fault repair on the manufacturing equipment through the desktop system operating software. The edge device automatically executes the corresponding operation steps through the simulated keyboard and mouse to repair the fault on the manufacturing equipment. Improve the efficiency of fault recovery, improve the normal operation efficiency of manufacturing equipment, and reduce dependence on central servers.

Description

Technical Field

[0001] The present invention belongs to the field of industrial manufacturing equipment, and in particular relates to an intelligent fault recovery method for industrial manufacturing equipment. Background Art

[0002] In the modern industrial manufacturing process, equipment fault recovery is a key link. Traditional fault recovery methods usually require technicians to conduct on-site inspections and repairs, which is not only time-consuming and laborious, but may also result in problems not being solved in a timely manner due to insufficient technicians. With the development of science and technology, edge computing and artificial intelligence technologies are gradually being applied to the fault recovery of industrial manufacturing equipment. Edge computing devices can collect and analyze equipment data in real time, while artificial intelligence technology can intelligently analyze this data to quickly identify and locate faults.

[0003] Existing solutions mainly collect and analyze real-time data from industrial manufacturing equipment through edge computing devices, and then transmit this data to a central server for further analysis and processing. The central server is usually equipped with advanced artificial intelligence algorithms that can intelligently analyze this data to identify and locate faults. Once the fault is identified and located, the central server sends instructions to the edge computing device, instructing it to take appropriate measures to fix the fault.

[0004] Although existing solutions can improve the fault recovery efficiency of industrial manufacturing equipment to a certain extent. However, this solution seems to be powerless in the face of certain specific situations. In particular, when the device or program lacks a direct remote command interface and cannot be controlled through the program, or cannot be connected to an external server for security reasons, and can only rely on manual operation at the machine console, the existing remote processing process cannot be directly applied. In addition, the existing solutions need to transmit a large amount of device data to the central server for processing, which not only requires a large amount of network bandwidth, but also may cause delays in data transmission. Secondly, the existing solutions rely on the computing power and algorithms of the central server. If the central server fails or is attacked, the entire system may be paralyzed. In addition, the existing solutions usually require technicians to conduct on-site inspections and repairs, which still consumes a lot of manpower and time. Therefore, the existing solutions still have certain limitations in practical applications. Summary of the invention

[0005] To achieve the above object, the technical solution of the present invention is as follows: an intelligent fault recovery method for industrial manufacturing equipment, comprising the following steps: Step 1: Edge device deployment; Step 2: data collection and analysis; Step 3: The fault is automatically repaired.

[0006] Based on the above technical features, edge device deployment: deploy an edge device outside the manufacturing equipment, which has computing and network communication capabilities. The edge device is simulated as a HID mouse and keyboard through the UDC driver, and the manufacturing equipment is operated through the simulated HID mouse and keyboard. At the same time, the edge device is connected to the video output of the manufacturing equipment through interfaces such as HDMI, VGA or DVI to obtain the desktop system screen of the manufacturing equipment.

[0007] Data collection and analysis: Edge devices collect relevant data of manufacturing equipment in real time, and analyze the acquired data through built-in AI algorithms to determine the corresponding fault information. AI algorithms can be based on machine learning, deep learning and other technologies, and have the ability of intelligent analysis and identification through learning and training of historical data.

[0008] Fault simulation repair: Based on the analysis results of the AI ​​algorithm, the edge device uses image recognition to find the corresponding simulated technician to perform fault repair directly on the manufacturing equipment through the desktop system operation software. Specifically, the edge device automatically executes the corresponding operation steps through the simulated keyboard and mouse to repair the fault on the manufacturing equipment.

[0009] As an improvement of the present invention, the step 1 edge device deployment includes the edge device simulating a HID mouse and keyboard through a USB interface to communicate with the manufacturing equipment, inserting one end of the HDMI data cable into the output port of the manufacturing equipment, inserting the other end of the HDMI data cable into the input port of the edge device, and adjusting the display settings of the device with HDMI input according to the configuration.

[0010] As an improvement of the present invention, the step 2 of data collection and analysis includes: Data collection: edge devices collect relevant data of manufacturing equipment in real time; Data flow, from the sensors of manufacturing equipment to the data acquisition modules of edge devices; AI algorithm module, which analyzes the acquired data through built-in AI algorithm to determine the corresponding fault information; Data analysis uses deep learning technology to learn and train historical data to have the ability of intelligent analysis and identification; Data analysis: After data collection, data analysis is performed through AI services on edge devices; Fault judgment: determine whether there is a fault based on the data analysis results.

[0011] As an improvement of the present invention, the step 3 of automatically repairing the fault includes: Image acquisition, taking real-time desktop screenshots from manufacturing equipment; Image preprocessing: perform necessary preprocessing on the screenshots; Feature extraction and matching: The application interface and operation function screens are limited in number, the annotated data is small, and the interface layout and content are relatively fixed. The pre-trained deep CNN model ResNet is used for transfer learning. Only the last layer needs to be replaced to perform classification tasks, and a small amount of data is used for fine-tuning. Determine the operation point, obtain the coordinate position and range of the upper left corner of the corresponding button in the video page based on the results of CNN prediction, and calculate the actual position to be clicked; Determine the current mouse position, and obtain the coordinates of the upper left corner of the mouse image in the video image based on the CNN prediction result; Simulate user interaction, control the simulated mouse to click at the corresponding coordinates through the HID interface, and trigger the application operation at the determined location; Verify the operation results, take another screenshot and analyze the relevant information in the new screenshot to confirm whether the status of the temperature switch has changed.

[0012] As an improvement of the present invention, the step of simulating user interaction in step three includes: Calculate the data of mouse movement, and calculate the relative coordinate position of the operating point relative to the current mouse position according to the current mouse position and the determined position of the operating point; Move the mouse. Open the HID device file example through the Python program installed on the edge device and send the command to move the mouse. If the relative position is far away, it will be split into multiple moves to the operation point coordinates. Click the mouse, and when it moves to the operation point coordinates, send the mouse click command.

[0013] As an improvement of the present invention, the image acquisition step in step three includes: Make sure / dev / video or similar device file represents the HDMI input device through the driver; Access the HDMI input video stream through the / dev / video device file, and use the video capture software to read and process the video stream; Get the desktop image of the current system through the acquired video.

[0014] As an improvement of the present invention, the algorithm module in step 2 includes using an LSTM network to capture long-term dependencies, including: The LSTM input layer defines the steps. Since we are focusing on multivariate time series prediction, the input shape is the number of samples, time steps, and number of features, where the number of features includes all the collected temperature, humidity, voltage, or current parameters. LSTM layer configuration step, adjust the number of LSTM units according to hardware performance, and consider whether to return the complete output sequence or only the output of the last time step; Output layer design steps: If the goal is to predict the temperature at a certain time in the future, use regression task settings, add a fully connected layer and use a linear activation function. If the goal is to classify abnormal situations, you can choose sigmoid or softmax activation functions; Compile the model step. For regression problems, the loss function is mean square error, and the optimizer is recommended to be Adam. For classification problems, the loss function is cross entropy. The fault judgment step compares the difference between the predicted value and the actual measured value to determine whether an abnormal situation has occurred.

[0015] As an improvement of the present invention, the fault judgment step in step 2 includes: The static threshold method sets a fixed error range. If the difference between the predicted value and the actual value exceeds this range, it is marked as an anomaly.

[0016] As an improvement of the present invention, the fault judgment step in step 2 includes: The dynamic threshold method takes into account the differences in normal fluctuations in different time periods and dynamically adjusts the threshold based on historical data, including setting it as a multiple of the standard deviation.

[0017] As an improvement of the present invention, the fault judgment step in step 2 includes: Residual analysis calculates the distribution characteristics of the differences between predicted values ​​and actual values, and considers differences that exceed a certain confidence level to be abnormal.

[0018] Compared with the prior art, the beneficial effects of the present invention are: 1. Reduce the number of on-site inspections by personnel: The present invention uses AI to analyze the manufacturing equipment data obtained by edge devices and determine the corresponding fault information, thereby reducing the number of on-site inspections by technicians. This not only saves human resources, but also reduces the risk of problems not being solved in a timely manner due to insufficient technicians.

[0019] 2. Improve fault recovery efficiency: The present invention uses edge devices to simulate technicians directly repairing faults through desktop system operating software on manufacturing equipment. This method can quickly locate and repair faults and reduce the time to recover from manufacturing equipment failures. At the same time, since the edge device can directly access the video output of the manufacturing equipment, it can obtain the desktop system screen of the manufacturing equipment in real time, allowing technicians to understand the fault situation more intuitively and repair it faster.

[0020] 3. Reduce the amount of data transmission: The present invention connects edge devices to manufacturing equipment, so that edge devices can directly obtain real-time data and desktop system images of manufacturing equipment, and analyze these data through AI to determine the corresponding fault information. This solution does not need to transmit a large amount of device data to the central server for processing, thereby greatly reducing the use of network bandwidth and improving the efficiency of data processing.

[0021] 4. Improve the normal operation efficiency of manufacturing equipment: Since the present invention can quickly locate and repair faults, it can reduce the abnormal operation time of manufacturing equipment, thereby improving the normal operation efficiency of manufacturing equipment. This not only improves production efficiency, but also reduces the maintenance cost of equipment.

[0022] 5. Reduce dependence on central servers: The present invention directly performs data analysis and fault repair through edge devices without relying on central servers. Therefore, even if the central server fails or is attacked, the entire system can still operate normally. This improves the reliability and security of the system. In summary, compared with the prior art, the present invention has the advantages of small data transmission volume, high fault recovery efficiency, fewer on-site personnel inspections, high normal operation efficiency of manufacturing equipment, and low dependence on central servers. Therefore, it has significant advantages in intelligent fault recovery of industrial manufacturing equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of the intelligent fault recovery method described in this embodiment. DETAILED DESCRIPTION

[0024] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0025] Embodiment: The explanations of the terms involved in this embodiment are shown in Table 1 below. Table 1

[0026] The overall method implementation flow chart of this embodiment is as follows Figure 1 As shown, specifically: Step 1: Edge device deployment Edge device: Use the core board model rk3588 or other similar devices. The device should have a GPU or NPU to implement AI computing and AI applications.

[0027] HID interface (mouse / keyboard): The edge device emulates a HID mouse and keyboard through the USB interface to communicate with the manufacturing equipment.

[0028] Video Output Interface (HDMI): Plug one end of the HDMI data cable into the output port of the manufacturing equipment. Plug the other end of the HDMI data cable into the input port of the edge device. Adjust the display settings (1920x1080 BGR3) of the device with HDMI input according to the configuration to ensure the video picture is clear and the color is accurate.

[0029] Step 2: Data collection and analysis Data collection: Edge devices collect relevant data of manufacturing equipment in real time, including parameters such as voltage, current, temperature, and humidity.

[0030] Data flow direction: from the sensors of manufacturing equipment (sensor 1, sensor 2, ...) to the data acquisition module of the edge device.

[0031] AI algorithm: The acquired data is analyzed through the built-in AI algorithm to determine the corresponding fault information.

[0032] AI algorithm module: uses deep learning technology and has the ability of intelligent analysis and recognition through learning and training of historical data.

[0033] Data analysis: After data collection, data analysis is performed through the AI ​​service of edge devices.

[0034] Fault diagnosis: Determine whether a fault exists based on the data analysis results.

[0035] The algorithm module steps are as follows: Algorithm module: LSTM (Long Short-Term Memory) is used to effectively capture long-term dependencies, which is particularly useful for identifying abnormal trends. LSTM input layer definition: Since we are concerned with multivariate time series prediction, the input shape should be (number of samples, time steps, number of features), where "number of features" includes all collected parameters (such as temperature, humidity, voltage, current, etc.).

[0036] LSTM layer configuration: adjust the number of LSTM units based on hardware performance, while taking into account whether to return the full output sequence (for multi-step predictions) or only the output of the last time step (for single-step predictions).

[0037] Output layer design: If the goal is to predict the temperature at a certain time in the future, use the regression task setting, add a fully connected layer and use a linear activation function; if the goal is to classify abnormal situations, you can choose to use a sigmoid or softmax activation function.

[0038] Compile model: For regression problems, the loss function can be the mean square error (MSE), and the recommended optimizer is Adam; for classification problems, the loss function is the cross entropy.

[0039] Training process: Use historical data sets for training, optimize hyperparameters through cross-validation and grid search, and ensure that the model maintains good generalization capabilities under different conditions.

[0040] Fault diagnosis: Compare the difference between the predicted value and the actual measured value to determine whether an abnormal situation has occurred. The main judgment methods are as follows: Static threshold method: Set a fixed error range (for example, ±2°C). If the difference between the predicted value and the actual value exceeds this range, it is marked as an anomaly.

[0041] Dynamic threshold method: Taking into account the differences in normal fluctuations in different time periods, the threshold can be dynamically adjusted based on historical data, for example, set to a multiple of the standard deviation.

[0042] Residual analysis: Calculates the distribution characteristics of the differences between predicted and actual values. Differences that exceed a certain confidence level (such as 95% or 99%) are considered abnormal.

[0043] Step 3: Automatic fault repair Troubleshooting Module: Image Acquisition: Get real-time desktop screenshots from manufacturing equipment.

[0044] Image preprocessing: Perform necessary preprocessing on the screenshots, such as resizing, grayscale conversion, noise removal, etc., to improve the accuracy of subsequent image recognition.

[0045] Application icon recognition: Use the trained model to recognize specific features of application icons in screenshots, locate and determine the position (coordinates) of each icon. It is recommended to use convolutional neural network (CNN) or template matching method.

[0046] Coordinate calculation: Based on the recognition results, calculate the precise coordinate position of the application icon that needs to be clicked.

[0047] Operation instruction generation: Generate corresponding mouse movement and click operation instructions based on the calculated coordinate information.

[0048] Operation execution: Through the simulated HID interface, control the mouse to click the icon of the corresponding coordinates to implement fault repair. For example, when the AI ​​algorithm recognizes that the CPU temperature of the manufacturing equipment is too high, the edge device will automatically perform cooling operations, such as reducing the fan speed or turning off part of the load.

[0049] The troubleshooting module steps specifically include: Image acquisition: Through a driver such as the V4L2 (Video for Linux 2) HDMI input support driver. Make sure that / dev / video or a similar device file represents the HDMI input device.

[0050] Access the video stream of HDMI input through the / dev / video device file. Use video capture software (such as ffmpeg, gstreamer, etc.) to read and process the video stream; Get the desktop image of the current system through the acquired video; Image preprocessing: resizing, grayscale conversion, noise removal, etc. to improve the accuracy of subsequent image recognition.

[0051] Feature extraction and matching: The number of application interfaces and operation function screens is limited, the data that can be annotated is small, and the interface layout and content are relatively fixed. The pre-trained deep CNN model ResNet can be used for transfer learning. Only the last layer (fully connected layer) needs to be replaced to perform classification tasks, and a small amount of data can be used for fine-tuning.

[0052] Determine the operation point: Based on the CNN prediction results, obtain the coordinate position and range of the upper left corner of the corresponding button in the video page, and calculate the actual position to be clicked. Generally, the center position of the identified button image is taken, and according to the predicted upper left corner coordinates (Top, Left) and the image size (Width, Height), the center position coordinates X = (Left + Width) / 2, Y = (Top + Height) / 2 can be calculated; Determine the current mouse position: Get the coordinates of the upper left corner of the mouse image in the video image based on the CNN prediction result; Simulate user interaction: Through the HID interface, control the simulated mouse to click at the corresponding coordinates and trigger the application operation at the determined position.

[0053] Calculate the mouse movement data: Based on the current mouse position and the determined position of the operating point, calculate the relative coordinate position of the operating point relative to the current mouse position. For example: the mouse coordinate position is (100, 100) and the operating point coordinate position is (200, 200), then the relative position is (100, 100).

[0054] Move the mouse: Open the HID device file such as ' / dev / hidg1' through the Python program installed on the edge device, and send the command to move the mouse. For example, if the relative position to be moved is (100,100) as described above, send (0,100,100). If the relative position is far away, it can be split into multiple moves to the operation point coordinates; Click the mouse: After moving to the operation point coordinates, send a mouse click command (1,0,0).

[0055] Verify the operation results: Take another screenshot and analyze the relevant information in the new screenshot to confirm whether the status of the temperature switch has changed.

[0056] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. An intelligent fault recovery method for industrial manufacturing equipment, characterized in that: The following steps are involved: Step 1: Edge device deployment; Step 2: data collection and analysis; Step 3: The fault is automatically repaired.

2. The intelligent fault recovery method for industrial manufacturing equipment according to claim 1 is characterized in that: The step 1 edge device deployment includes the edge device simulating a HID mouse and keyboard through a USB interface to communicate with the manufacturing device, inserting one end of the HDMI data cable into the output port of the manufacturing device, inserting the other end of the HDMI data cable into the input port of the edge device, and adjusting the display settings of the device with HDMI input according to the configuration.

3. The intelligent fault recovery method for industrial manufacturing equipment according to claim 1 is characterized in that: The step 2 of data collection and analysis includes: Data collection: edge devices collect relevant data of manufacturing equipment in real time; Data flow, from the sensors of manufacturing equipment to the data acquisition modules of edge devices; AI algorithm module, which analyzes the acquired data through built-in AI algorithm to determine the corresponding fault information; Data analysis uses deep learning technology to learn and train historical data to have the ability of intelligent analysis and identification; Data analysis: After data collection, data analysis is performed through AI services on edge devices; Fault judgment: determine whether there is a fault based on the data analysis results.

4. The intelligent fault recovery method for industrial manufacturing equipment according to claim 1, characterized in that: The step 3 of automatic fault repair includes: Image acquisition, taking real-time desktop screenshots from manufacturing equipment; Image preprocessing: perform necessary preprocessing on the screenshots; Feature extraction and matching: The application interface and operation function screens are limited in number, the annotated data is small, and the interface layout and content are relatively fixed. The pre-trained deep CNN model ResNet is used for transfer learning. Only the last layer needs to be replaced to perform classification tasks, and a small amount of data is used for fine-tuning. Determine the operation point, obtain the coordinate position and range of the upper left corner of the corresponding button in the video page based on the results of CNN prediction, and calculate the actual position to be clicked; Determine the current mouse position, and obtain the coordinates of the upper left corner of the mouse image in the video image based on the CNN prediction result; Simulate user interaction, control the simulated mouse to click at the corresponding coordinates through the HID interface, and trigger the application operation at the determined location; Verify the operation results, take another screenshot and analyze the relevant information in the new screenshot to confirm whether the status of the temperature switch has changed.

5. The intelligent fault recovery method for industrial manufacturing equipment according to claim 4 is characterized in that: The step of simulating user interaction in step 3 includes: Calculate the data of mouse movement, and calculate the relative coordinate position of the operating point relative to the current mouse position according to the current mouse position and the determined position of the operating point; Move the mouse. Open the HID device file example through the Python program installed on the edge device and send the command to move the mouse. If the relative position is far away, it will be split into multiple moves to the operation point coordinates. Click the mouse, and when it moves to the operation point coordinates, send the mouse click command.

6. The intelligent fault recovery method for industrial manufacturing equipment according to claim 4, characterized in that: The image acquisition step in step 3 includes: Make sure / dev / video or similar device file represents the HDMI input device through the driver; Access the HDMI input video stream through the / dev / video device file, and use the video capture software to read and process the video stream; Get the desktop image of the current system through the acquired video.

7. The intelligent fault recovery method for industrial manufacturing equipment according to claim 3 is characterized in that: The algorithm module in step 2 includes using an LSTM network to capture long-term dependencies, including: The LSTM input layer defines the steps. Since we are focusing on multivariate time series prediction, the input shape is the number of samples, time steps, and number of features, where the number of features includes all the collected temperature, humidity, voltage, or current parameters. LSTM layer configuration step, adjust the number of LSTM units according to hardware performance, and consider whether to return the complete output sequence or only the output of the last time step; Output layer design steps: If the goal is to predict the temperature at a certain time in the future, use regression task settings, add a fully connected layer and use a linear activation function. If the goal is to classify abnormal situations, you can choose sigmoid or softmax activation functions; Compile the model. For regression problems, the loss function is mean square error, and the optimizer is Adam. For classification problems, the loss function is cross entropy. The fault judgment step compares the difference between the predicted value and the actual measured value to determine whether an abnormal situation has occurred.

8. The intelligent fault recovery method for industrial manufacturing equipment according to claim 7, characterized in that: The fault judgment step in step 2 includes: The static threshold method sets a fixed error range. If the difference between the predicted value and the actual value exceeds this range, it is marked as an anomaly.

9. The intelligent fault recovery method for industrial manufacturing equipment according to claim 7, characterized in that: The fault judgment step in step 2 includes: The dynamic threshold method takes into account the differences in normal fluctuations in different time periods and dynamically adjusts the threshold based on historical data, including setting it as a multiple of the standard deviation.

10. The intelligent fault recovery method for industrial manufacturing equipment according to claim 7, characterized in that: The fault judgment step in step 2 includes: Residual analysis calculates the distribution characteristics of the differences between predicted values ​​and actual values, and considers differences that exceed a certain confidence level to be abnormal.

Images