Non-standard machine automation operation method, device and equipment and storage medium

Abstract

The present application relates to the technical field of semiconductor manufacturing, and particularly relates to a method, device and equipment for realizing automatic operation of non-standard machine and a storage medium, wherein a first screen image of the non-standard machine is periodically collected by using a KVM hardware device, the first screen image is preprocessed, a preset OCR recognition model is used to perform visual recognition on the preprocessed image, machine information is extracted according to the recognition result, the machine information is converted into a standard industrial protocol format, and is reported to an upper control system, so that the upper control system generates and issues a control instruction according to the machine information, a corresponding keyboard and mouse operation sequence is generated according to the control instruction, the keyboard and mouse operation sequence is executed by using the KVM hardware device, a second screen image of the non-standard machine is collected again after the control is executed, a state change of the non-standard machine is analyzed according to the second screen image, and state change information is generated, and the automatic operation method has strong compatibility, is easy to deploy and does not need to be deeply modified.

Description

Technical Field

[0001] This invention relates to the field of semiconductor manufacturing technology, and in particular to a method, apparatus, equipment and storage medium for automating non-standard machine operations. Background Technology

[0002] In the modern semiconductor manufacturing industry, to achieve efficient and precise production control, information management systems based on CIM architecture are widely adopted. These systems, such as EAP and MES, collaborate with the equipment control layer to achieve full-process automation. However, a large number of traditional non-standard equipment still exist in the industry. Due to their high degree of customization and significant structural differences, they are difficult to directly integrate into standard automation systems, and most still rely on manual operation, resulting in low efficiency, large errors, and safety hazards. Existing automation solutions are mostly designed for general standard equipment and cannot flexibly adapt to the diverse interfaces and interfaces of non-standard equipment. Modifying the equipment's hardware and software, on the other hand, involves long development cycles, high costs, and poor portability. Therefore, there is an urgent need for an automated operation method that is highly compatible, easy to deploy, and does not require deep modification of non-standard equipment to improve its production efficiency and system integration capabilities. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a method, apparatus, equipment and storage medium for automating non-standard machine operations that is highly compatible, easy to deploy and does not require deep modification of non-standard equipment.

[0004] The first aspect of this invention provides a method for automating the operation of a non-standard machine, comprising: acquiring interface layout information and protocol communication port information of the non-standard machine; connecting a KVM hardware device to the non-standard machine based on the protocol communication port information; periodically acquiring a first screen image of the non-standard machine using the KVM hardware device based on the interface layout information; performing image preprocessing on the first screen image to obtain a preprocessed image; performing visual recognition on the preprocessed image using a preset OCR recognition model to obtain a recognition result; extracting machine information based on the recognition result, wherein the machine information is one or more of parameter value information, operating status information, and alarm information; converting the machine information into a standard industrial protocol format and reporting it to a host control system, so that the host control system generates and issues control commands based on the machine information, and generates a corresponding keyboard and mouse operation sequence based on the control commands; executing the keyboard and mouse operation sequence using the KVM hardware device; acquiring a second screen image of the non-standard machine again after executing the control; analyzing the state changes of the non-standard machine based on the second screen image and generating state change information.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the step of acquiring the interface layout information and protocol communication port information of the non-standard machine, connecting the KVM hardware device to the non-standard machine based on the protocol communication port information, and periodically collecting the first screen image of the non-standard machine using the KVM hardware device based on the interface layout information includes: acquiring the interface layout information and protocol communication port information of the non-standard machine; connecting the KVM hardware device to the non-standard machine based on the protocol communication port information; establishing a screenshot triggering mechanism, the screenshot triggering mechanism including active periodic triggering and passive alarm triggering; and collecting the first screen image of the non-standard machine using the KVM hardware device based on the screenshot triggering mechanism and the interface layout information.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the step of preprocessing the first screen image to obtain a preprocessed image includes: converting the first screen image into a grayscale image and performing size normalization processing on the grayscale image; performing median filtering processing on the grayscale image to suppress small reflective point noise; segmenting the reflective region of the grayscale image based on a preset grayscale threshold; extracting the contour of the reflective region; and calculating the area information of the reflective region based on the contour; when the area information is less than or equal to a preset area threshold, replacing the grayscale value of the reflective region with the neighborhood grayscale mean; when the area information is greater than the preset area threshold, repairing the reflective region using an image inpainting function; performing grayscale normalization processing on the repaired grayscale image; and performing adaptive Gaussian filtering processing on the grayscale image, and then... The gradient operator calculates the image gradient of the grayscale image to obtain a gradient image. The gradient image is then fused with the grayscale image to enhance the text edge contours, resulting in an enhanced image. An adaptive thresholding method is used to calculate the global optimal segmentation threshold and perform preliminary binarization on the enhanced image to obtain a preliminary binary image containing the parametric text. The preliminary binary image is divided into multiple sub-regions, and a local optimal threshold is calculated for each sub-region. When the deviation between the local optimal threshold and the global optimal segmentation threshold exceeds a preset range, the sub-region is re-binarized using the local optimal threshold to obtain a final binary image. Morphological optimization is performed on the final binary image to obtain a preprocessed image. The proportion of text pixels in the preprocessed image is statistically analyzed. When the proportion of text pixels is lower than a preset proportion threshold, adaptive Gaussian filtering is performed.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the step of visually recognizing the preprocessed image using a preset OCR recognition model to obtain a recognition result, and extracting machine information based on the recognition result, wherein the machine information is one or more of parameter value information, operating status information, and alarm information, includes: constructing a three-level time-series database based on process stage, equipment number, and time dimension; visually recognizing the preprocessed image using a preset OCR recognition model to obtain a recognition result; extracting machine information based on the recognition result, wherein the machine information is one or more of parameter value information, operating status information, and alarm information; binding the machine information with a corresponding timestamp, and storing the machine information in the three-level time-series database.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the step of converting the machine information into a standard industrial protocol format and reporting it to the upper control system so that the upper control system can generate and issue control commands based on the machine information, and generate a corresponding keyboard and mouse operation sequence based on the control commands, includes: converting the machine information into a standard industrial protocol format and reporting it to the upper control system so that the upper control system can generate and issue control commands based on the machine information; receiving the control commands issued by the upper control system and parsing the control commands; and generating a corresponding keyboard and mouse operation sequence based on the parsed control commands.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of using the KVM hardware device to execute the keyboard and mouse operation sequence, and after executing the control, acquiring the second screen image of the non-standard machine again, analyzing the state changes of the non-standard machine based on the second screen image, and generating state change information includes: using the KVM hardware device to execute the keyboard and mouse operation sequence, and after executing the control, acquiring the second screen image of the non-standard machine again; comparing the second screen image with the first screen image to obtain comparison information; analyzing the state changes of the non-standard machine based on the comparison information and generating state change information; determining whether the non-standard machine has reached the expected state based on the state change information; if not, re-executing the keyboard and mouse operation sequence; and automatically stopping the operation and generating a maintenance notification when the expected state is not reached after a preset number of consecutive executions.

[0010] Optionally, in the sixth implementation of the first aspect of the present invention, after executing the keyboard and mouse operation sequence using the KVM hardware device, acquiring the second screen image of the non-standard machine again after executing the control, analyzing the state changes of the non-standard machine based on the second screen image, and generating state change information, the method further includes: temporally associating the machine information corresponding to the continuously acquired first screen images according to timestamps, and constructing a dynamic trend curve of core process parameters using a time-series analysis algorithm; establishing a correlation mapping model between parameters; when the alarm information is identified, locating a predetermined time window before and after the alarm occurrence time in the time-series database; automatically tracing back the parameter change trajectory in the dynamic trend curve within the time window, using the correlation mapping model to locate the root cause of the anomaly based on the parameter change trajectory, and identifying the deviation parameter; generating a time-series traceability report based on the alarm occurrence time, the dynamic trend curve, the root cause of the anomaly, and the deviation parameter; determining the parameter type of the deviation parameter, dynamically adjusting the image enhancement algorithm parameters corresponding to the parameter type, and optimizing the OCR recognition model using the image enhancement algorithm parameters.

[0011] A second aspect of the present invention provides an automated operation device for non-standard machines, comprising: an acquisition and connection module for acquiring interface layout information and protocol communication port information of the non-standard machine, connecting a KVM hardware device to the non-standard machine based on the protocol communication port information, and periodically acquiring a first screen image of the non-standard machine based on the interface layout information using the KVM hardware device; a preprocessing module for performing image preprocessing on the first screen image to obtain a preprocessed image; and a recognition and extraction module for performing visual recognition on the preprocessed image using a preset OCR recognition model to obtain a recognition result, and extracting machine data based on the recognition result. The system includes: a machine information module, which is one or more of parameter value information, operating status information, and alarm information; a conversion and reporting generation module, which converts the machine information into a standard industrial protocol format and reports it to the upper control system, so that the upper control system can generate and issue control commands based on the machine information, and generate corresponding keyboard and mouse operation sequences based on the control commands; and an execution acquisition and analysis generation module, which uses the KVM hardware device to execute the keyboard and mouse operation sequences, and after executing the control, acquires the second screen image of the non-standard machine again, analyzes the status changes of the non-standard machine based on the second screen image, and generates status change information.

[0012] Optionally, in a first implementation of the second aspect of the present invention, the acquisition and connection acquisition module includes: an acquisition unit for acquiring interface layout information and protocol communication port information of the non-standard machine; a connection unit for connecting the KVM hardware device to the non-standard machine based on the protocol communication port information; a setup unit for setting up a screenshot triggering mechanism, the screenshot triggering mechanism including active periodic triggering and passive alarm triggering; and an acquisition unit for acquiring a first screen image of the non-standard machine using the KVM hardware device based on the screenshot triggering mechanism and the interface layout information.

[0013] Optionally, in a second implementation of the second aspect of the present invention, the preprocessing module includes: a conversion and standardization unit, configured to convert the first screen image into a grayscale image and perform size standardization processing on the grayscale image; a filtering, segmentation, extraction, and calculation unit, configured to perform median filtering on the grayscale image to suppress small reflective noise, segment the reflective region of the grayscale image based on a preset grayscale threshold, extract the contour of the reflective region, and calculate the area information of the reflective region based on the contour; a replacement and repair unit, configured to replace the grayscale value of the reflective region with the neighborhood grayscale mean when the area information is less than or equal to a preset area threshold, and repair the reflective region using an image repair function when the area information is greater than the preset area threshold; a normalization unit, configured to perform grayscale normalization processing on the repaired grayscale image; and a filtering, calculation, and fusion unit, configured to perform adaptive Gaussian filtering on the grayscale image and perform gradient descent. A gradient operator calculates the image gradient of the grayscale image to obtain a gradient image. The gradient image is then fused with the grayscale image to enhance the text edge contours, resulting in an enhanced image. A binarization unit is used to calculate the global optimal segmentation threshold using an adaptive thresholding method and perform preliminary binarization processing on the enhanced image to obtain a preliminary binary image containing the parameterized text. A segmentation binarization unit is used to divide the preliminary binary image into multiple sub-regions and calculate the local optimal threshold for each sub-region. When the deviation between the local optimal threshold and the global optimal segmentation threshold exceeds a preset range, the sub-region is re-binarized using the local optimal threshold to obtain a final binary image. An optimization and statistical return unit is used to perform morphological optimization processing on the final binary image to obtain a preprocessed image. The proportion of text pixels in the preprocessed image is statistically analyzed. When the proportion of text pixels is lower than a preset proportion threshold, the process returns to perform adaptive Gaussian filtering.

[0014] Optionally, in a third implementation of the second aspect of the present invention, the identification and extraction module includes: a construction unit, used to construct a three-level time-series database based on process stage, equipment number, and time dimension; an identification unit, used to perform visual identification on the preprocessed image using a preset OCR identification model to obtain identification results; an extraction unit, used to extract machine information based on the identification results, wherein the machine information is one or more of parameter value information, operating status information, and alarm information; and a binding and storage unit, used to bind the machine information with a corresponding timestamp and store the machine information in the three-level time-series database.

[0015] Optionally, in a fourth implementation of the second aspect of the present invention, the conversion and reporting generation module includes: a conversion and reporting unit, used to convert the machine information into a standard industrial protocol format and report it to the upper control system, so that the upper control system can generate and issue control commands based on the machine information; a receiving and parsing unit, used to receive the control commands issued by the upper control system and parse the control commands; and a generation unit, used to generate a corresponding keyboard and mouse operation sequence based on the parsed control commands.

[0016] Optionally, in a fifth implementation of the second aspect of the present invention, the execution acquisition, analysis, and generation module includes: an execution acquisition unit, configured to execute the keyboard and mouse operation sequence using the KVM hardware device, and acquire the second screen image of the non-standard machine again after execution control; a comparison unit, configured to compare the second screen image with the first screen image to obtain comparison information; an analysis and generation unit, configured to analyze the state changes of the non-standard machine based on the comparison information and generate state change information; a judgment unit, configured to judge whether the non-standard machine has reached the expected state based on the state change information; an execution unit, configured to re-execute the keyboard and mouse operation sequence if not; and a stop generation unit, configured to automatically stop the execution operation and generate a maintenance notification when the expected state is not reached after a preset number of consecutive executions.

[0017] Optionally, in the sixth implementation of the second aspect of the present invention, it further includes: an association construction module, used to perform time-series association of the machine information corresponding to the continuously acquired first screen images according to timestamps, and to construct a dynamic trend curve of the core process parameters using a time-series analysis algorithm; an establishment module, used to establish an association mapping model between parameters; a positioning module, used to locate a predetermined time window before and after the alarm occurrence time in the time-series database when the alarm information is identified; a backtracking positioning and identification module, used to automatically backtrack the parameter change trajectory in the dynamic trend curve within the time window, use the association mapping model to locate the root cause of the anomaly based on the parameter change trajectory, and identify the deviation parameter; a generation module, used to generate a time-series tracing report based on the alarm occurrence time, the dynamic trend curve, the root cause of the anomaly, and the deviation parameter; and a determination adjustment and optimization module, used to determine the parameter type of the deviation parameter, dynamically adjust the image enhancement algorithm parameters corresponding to the parameter type, and optimize the OCR recognition model using the image enhancement algorithm parameters.

[0018] A third aspect of the present invention provides an automated operation device for non-standard machines, the automated operation device for non-standard machines comprising: a memory and at least one processor, the memory storing instructions; at least one processor calling the instructions in the memory to cause the automated operation device for non-standard machines to perform the various steps of the automated operation method for non-standard machines described above.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the method for automating non-standard machine operations as described above.

[0020] In the technical solution of this invention, the first screen image of a non-standard machine is periodically acquired using KVM hardware. The first screen image is preprocessed, and a preset OCR recognition model is used to perform visual recognition on the preprocessed image. Machine information is extracted based on the recognition results, converted into a standard industrial protocol format, and reported to the upper control system. The upper control system then generates and issues control commands based on the machine information. The corresponding keyboard and mouse operation sequence is generated based on the control commands, and the keyboard and mouse operation sequence is executed using KVM hardware. After the control is executed, the second screen image of the non-standard machine is acquired again. The state changes of the non-standard machine are analyzed based on the second screen image, and state change information is generated. This method is highly compatible, easy to deploy, and does not require deep modification of the non-standard equipment. Attached Figure Description

[0021] Figure 1 This is a first flowchart of a method for automating non-standard machine operations provided in an embodiment of the present invention;

[0022] Figure 2 This is a second flowchart of a method for automating non-standard machine operations provided in an embodiment of the present invention;

[0023] Figure 3 This is a third flowchart of a method for automating non-standard machine operations provided in an embodiment of the present invention;

[0024] Figure 4 This is a fourth flowchart of a method for automating non-standard machine operations provided in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of a structure for an automated non-standard machine tool operation device provided in an embodiment of the present invention;

[0026] Figure 6 This is another structural schematic diagram of the device for automating non-standard machine operations provided in an embodiment of the present invention;

[0027] Figure 7 This is a structural diagram of an automated non-standard machine tool provided in an embodiment of the present invention. Detailed Implementation

[0028] This invention provides a method, apparatus, equipment, and storage medium for automating non-standard machine operations, which is highly compatible, easy to deploy, and does not require extensive modification of non-standard equipment.

[0029] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the method for automating non-standard machine operations in this invention includes:

[0031] 101. Obtain the interface layout information and protocol communication port information of the non-standard machine. Based on the protocol communication port information, connect the KVM hardware device to the non-standard machine. Based on the interface layout information, use the KVM hardware device to periodically collect the first screen image of the non-standard machine.

[0032] In this embodiment, a non-standard machine refers to semiconductor manufacturing equipment that does not integrate standard communication interfaces (such as SECS / GEM, OPC UA). Its operating interface is often a touchscreen or industrial computer screen, which cannot directly communicate with the upper-level control system. A KVM hardware device is an external hardware that supports video capture and simulated keyboard and mouse input. It connects to the display output port of the non-standard machine through video interfaces such as VGA / HDMI, and simulates keyboard and mouse operations via USB or serial port, achieving non-intrusive monitoring and control of the machine. The system first identifies the device model and communication interface of the target non-standard machine, obtains its interface layout information (such as screen resolution, button positions, status display area, etc.) and supported communication port information (such as VGA / HDMI, etc.). The system uses various interfaces such as VGA, USB, and network interfaces to connect KVM hardware devices to non-standard machine hosts via video cables (e.g., VGA splitters) and keyboard / mouse cables (USB interfaces) based on protocol communication port information. This ensures synchronous transmission of video and control signals. The KVM hardware devices are connected to automated control devices via network interfaces to achieve remote control and image acquisition. The system periodically acquires screen images of non-standard machines based on interface layout information using the KVM hardware devices according to a preset screenshot triggering mechanism (e.g., periodic triggering or alarm linkage triggering). These images serve as the first screen image for subsequent status recognition and control.

[0033] 102. Perform image preprocessing on the first screen image to obtain a preprocessed image;

[0034] In this embodiment, the recognition effect of the first screen image may be affected by problems such as device screen reflection, afterimages, and noise. Therefore, image preprocessing is required for the first screen image. The preprocessing process includes: converting the acquired RGB image to grayscale to reduce the amount of computation; performing size standardization to adapt to the input requirements of the OCR model; applying medium-level filtering to eliminate salt-and-pepper noise; identifying reflective areas by grayscale threshold segmentation, and selecting neighborhood grayscale replacement or image restoration algorithms for local correction based on the area size; performing grayscale normalization to enhance contrast; then using adaptive Gaussian filtering to smooth the image, combining gradient operators to extract text edges, and fusing gradient images and grayscale images to enhance contours; finally, performing global and local adaptive threshold binarization processing, combined with morphological operations to optimize character connectivity, to obtain a clear, low-noise preprocessed image.

[0035] 103. Use a preset OCR recognition model to perform visual recognition on the preprocessed image to obtain the recognition result. Extract the machine information based on the recognition result. The machine information is one or more of the following: parameter value information, operating status information, and alarm information.

[0036] In this embodiment, the preset OCR recognition model is a deep learning-based text detection and recognition model, such as CRNN or DBNet. Trained on a large number of industrial screen images, it can adapt to text recognition with different fonts, sizes, and tilt angles. The system calls the preset OCR recognition model to perform visual recognition on the preprocessed image. The visual recognition process includes: performing text detection on the preprocessed image and locating parameter labels and numerical regions; performing character recognition on each text region and outputting the recognition result string; based on the predefined correspondence between parameter labels and numerical values, the system extracts parameter value information (such as temperature, pressure, power, speed, etc.), operating status information (such as running, stopped, alarm), and alarm information (such as error codes and alarm descriptions) from the recognition result to obtain the recognition result. The system parses the recognition result into structured machine information according to the pre-configured interface template and region positioning rules, binds it with the collection timestamp, and stores it in a time-series database to form a traceable data record.

[0037] 104. Convert the machine information into a standard industrial protocol format and report it to the upper control system so that the upper control system can generate and issue control commands based on the machine information, and generate corresponding keyboard and mouse operation sequences based on the control commands.

[0038] In this embodiment, the extracted machine information needs to be converted into a standard industrial protocol format, such as JSON, XML, or a message format specific to SECS / GEM, and reported to the upper control system (such as MES or EAP) via a network protocol so that the upper control system can parse and process it. The upper control system performs logical judgment and scheduling decisions based on the received machine information and generates corresponding control commands (such as start, stop, parameter adjustment, etc.). After the control commands are sent to this system, the system parses them into a series of keyboard and mouse operation sequences, such as moving the cursor to a parameter input box, clicking, inputting a value, pressing the confirmation key, etc. These operation sequences will be executed by the KVM hardware device to simulate manual operation.

[0039] 105. Use KVM hardware to execute keyboard and mouse operation sequences. After executing the control, collect the second screen image of the non-standard machine again. Analyze the status changes of the non-standard machine based on the second screen image and generate status change information.

[0040] In this embodiment, the KVM hardware device executes the generated keyboard and mouse operation sequence to complete the control operation of the non-standard machine. After the operation is executed, the system again acquires the machine screen image through the KVM device as the second screen image. By comparing the second screen image with the first screen image, the system analyzes whether the machine status has changed as expected, such as whether the parameters have been successfully modified, whether the alarm has been cleared, and whether the interface has changed. Based on the comparison of image difference, region matching, or OCR recognition results, the system generates status change information and determines whether the control has achieved the expected effect. If not, a retry mechanism can be triggered or an anomaly can be reported, thereby realizing closed-loop control and status verification.

[0041] In this embodiment of the invention, KVM hardware is used to periodically collect images of non-standard machine screens and simulate keyboard and mouse operations. Combined with image preprocessing, OCR recognition, protocol conversion, and state change analysis, a complete closed loop for automated operation of non-standard machines is constructed. This method achieves real-time monitoring and remote control of the operating status of non-standard machines without modifying the equipment hardware or relying on dedicated communication interfaces. It significantly improves the automation level of old equipment in the semiconductor manufacturing process and has the advantages of strong compatibility, low deployment cost, and rapid adaptation to various machine interfaces.

[0042] Please see Figure 2 In the second embodiment of the method for automating non-standard machine operations according to the present invention, steps 101 and 102 include:

[0043] 201. Obtain the interface layout information and protocol communication port information of non-standard machines;

[0044] In this embodiment, the system obtains the interface layout information and protocol communication port information of the non-standard machine through the device configuration file or manual configuration interface. The interface layout information includes screen resolution, coordinates of each functional area of ​​the screen, parameter display position, button control position, etc.; the protocol communication port information includes the video interface (such as HDMI, VGA) and keyboard and mouse simulation interface (such as USB, PS / 2) of the KVM device connecting to the machine.

[0045] 202. Connect KVM hardware devices to non-standard machines based on protocol communication port information;

[0046] In this embodiment, based on the obtained protocol communication port information, the KVM hardware device is connected to the display output port of the machine via a video cable, and connected to the machine's USB port via a USB cable or a serial-to-USB adapter is used to simulate keyboard and mouse input, thereby realizing screen signal acquisition and remote control capabilities. After the physical connection is completed, the device driver will be automatically loaded, and the system can capture video streams and send simulated keyboard and mouse commands.

[0047] 203. Establish a screenshot triggering mechanism, which includes active periodic triggering and passive alarm triggering;

[0048] In this embodiment, active periodic triggering means that the system automatically captures screen images at preset time intervals (such as every 5 seconds); passive alarm triggering means that when the system recognizes alarm information on the screen through OCR, it immediately triggers a screenshot to capture the alarm interface for subsequent analysis and processing.

[0049] 204. Based on the screenshot triggering mechanism and interface layout information, use KVM hardware to capture the first screen image of a non-standard machine.

[0050] In this embodiment, the system determines the screenshot area (such as full screen or parameter display area) based on the interface layout information, and controls the KVM device to perform the screenshot operation in conjunction with the trigger mechanism, saving it as a bitmap or video stream frame as the first screen image for subsequent processing.

[0051] 205. Convert the first screen image to a grayscale image and perform size normalization on the grayscale image;

[0052] In this embodiment, the system uses an image processing library to convert the acquired RGB image into a grayscale image to eliminate color interference. The grayscale conversion uses a weighted average method to retain brightness information. Size standardization scales the image to a fixed resolution (e.g., 1920×1080) to ensure consistency of screenshots from different devices, which facilitates subsequent algorithm processing and model input.

[0053] 206. Perform median filtering on the grayscale image to suppress small reflective noise, segment the reflective area of ​​the grayscale image based on a preset grayscale threshold, extract the contour of the reflective area, and calculate the area information of the reflective area based on the contour.

[0054] In this embodiment, median filtering is performed on the grayscale image. The median filtering uses a 3×3 or 5×5 window to effectively remove isolated reflective noise points on the screen initially. Then, the reflective area is segmented based on a high grayscale threshold (e.g., ≥230). The contour of the connected region is extracted by combining morphological operations, and the area of ​​the contour pixels is calculated as the area information of the reflective area.

[0055] 207. When the area information is less than or equal to the preset area threshold, the gray value of the reflective area is replaced by the average gray value of the neighborhood. When the area information is greater than the preset area threshold, the reflective area is repaired using an image restoration function.

[0056] In this embodiment, when the area information is less than or equal to a preset area threshold (e.g., 100 pixels), smooth repair is achieved by replacing the average of neighboring pixels; when the area information is greater than the preset area threshold, an image repair algorithm (e.g., a repair model based on texture synthesis) is used to reconstruct the content to avoid information loss.

[0057] 208. Perform grayscale normalization on the repaired grayscale image;

[0058] In this embodiment, the repaired grayscale image is subjected to grayscale normalization processing. Grayscale normalization is achieved by linear stretching or histogram equalization, which adjusts the grayscale range of the image to a standard range (such as 0-255) to enhance the overall contrast of the image.

[0059] 209. Perform adaptive Gaussian filtering on the grayscale image, and calculate the image gradient of the grayscale image using the gradient operator to obtain the gradient image. Then, fuse the gradient image with the grayscale image to enhance the text edge contours and obtain the enhanced image.

[0060] In this embodiment, the grayscale image is processed by adaptive Gaussian filtering. The adaptive Gaussian filtering adjusts the filtering intensity according to the local grayscale variance, smoothing noise while preserving edges. The image gradient of the grayscale image is calculated by a gradient operator to obtain a gradient image. The gradient operator uses the Sobel or Canny operator to extract edge information. The gradient image and the grayscale image are fused by weighted superposition to highlight the outline of the text region, resulting in an enhanced image.

[0061] 210. The adaptive thresholding method is used to calculate the global optimal segmentation threshold and perform preliminary binarization processing on the enhanced image to obtain a preliminary binary image containing the parametric text;

[0062] In this embodiment, an adaptive thresholding method (such as the Otsu method or the local adaptive thresholding method) is used to determine the optimal segmentation threshold based on the image grayscale distribution. The enhanced image is then converted into a black and white binary image, with the text area being white (foreground) and the background being black. This allows the text to be separated from the background, resulting in a preliminary binary image.

[0063] 211. Divide the initial binary image into multiple sub-regions, calculate the local optimal threshold for each sub-region, and when the deviation between the local optimal threshold and the global optimal segmentation threshold exceeds the preset range, use the local optimal threshold to re-binarize the sub-region to obtain the final binary image.

[0064] In this embodiment, to further improve the binarization accuracy, the system divides the initial binary image into multiple sub-regions. The sub-region division adopts the grid method or connected component segmentation. The local optimal threshold of each sub-region is calculated. The local threshold calculation adopts the local Otsu method or the mean method. When the deviation between the local optimal threshold and the global optimal segmentation threshold exceeds the preset range, the sub-region is re-binarized using the local optimal threshold to obtain the final binary image.

[0065] 212. Perform morphological optimization on the final binary image to obtain a preprocessed image. Calculate the proportion of text pixels in the preprocessed image. When the proportion of text pixels is lower than the preset proportion threshold, return to perform adaptive Gaussian filtering.

[0066] In this embodiment, morphological optimization processing is performed on the final binary image. Morphological optimization includes opening operation to remove isolated noise points and closing operation to connect broken characters to obtain a preprocessed image. Then, the proportion of text pixels in the preprocessed image is counted. When the proportion of text pixels is lower than the preset proportion threshold, it indicates that the image quality is poor. In this case, the image is returned to perform adaptive Gaussian filtering processing again to ensure the OCR recognition effect.

[0067] In this embodiment of the invention, by acquiring the machine interface layout and communication port information, a screenshot triggering mechanism combining active and passive methods is established. A multi-level image preprocessing process is designed to address common interferences in industrial screen images (such as reflection, noise, and uneven lighting), including grayscale conversion, median filtering, adaptive repair of reflective areas, gradient enhancement, adaptive threshold binarization, and morphological optimization. This preprocessing method effectively improves the clarity and recognizability of text areas, laying a solid foundation for subsequent high-precision OCR recognition and significantly enhancing the robustness and information extraction accuracy of the system in complex industrial environments.

[0068] Please see Figure 3 In the third embodiment of the method for automating non-standard machine operations in this invention, steps 103 and 104 include:

[0069] 301. Construct a three-level time-series database based on process stage, equipment number, and time dimension;

[0070] In this embodiment, the three-level time-series database adopts a hierarchical structure design to adapt to the data management needs of multiple machines and processes in semiconductor manufacturing. The first level is divided according to process stage (such as photolithography, etching, thin film deposition), the second level distinguishes different machines under the same process according to equipment number, and the third level is the time dimension, recording data with timestamps as indexes. This three-level time-series database is constructed using time-series databases (such as InfluxDB and TimescaleDB), which can efficiently store and query serialized data with timestamps, and support rapid retrieval and analysis by process, equipment, and time range, providing a structured data foundation for subsequent trend analysis and anomaly tracing.

[0071] 302. Use a preset OCR recognition model to perform visual recognition on the preprocessed image to obtain the recognition result;

[0072] In this embodiment, the preset OCR recognition model is an end-to-end text recognition model based on an attention mechanism (such as SVTR or ABINet). This OCR recognition model incorporates a large number of industrial screen text images during the training phase, and has the ability to resist interference and deformation. The visual recognition process includes: performing feature extraction and text line detection on the preprocessed image to locate all text regions; performing sequence recognition on each text region and outputting the corresponding text string; and saving the recognition results in a structured format (such as a dictionary or JSON), which includes information such as text content, confidence level, and location coordinates, providing raw data for subsequent information extraction.

[0073] 303. Extract machine information based on the identification results. Machine information includes one or more of the following: parameter value information, operating status information, and alarm information.

[0074] In this embodiment, the system parses and extracts key machine information from the recognition results based on predefined rule templates or regular expressions. Parameter value information is matched with label numerical patterns (e.g., temperature: 25.5℃), operating status information is matched with fixed status words (e.g., running, idle, alarm), and alarm information is matched with error codes and descriptive text (e.g., ERR001: abnormal air pressure). The extracted information is classified and stored, and associated with its corresponding screen position and timestamp to form usable monitoring data units.

[0075] 304. Bind the machine information to the corresponding timestamp and store the machine information in the three-level time series database;

[0076] In this embodiment, the system records a precise timestamp when capturing each frame of screen image. After extracting the machine information, the information entries are bound to the timestamps to form key-value pairs of timestamps and information. Subsequently, the data is categorized according to the equipment number and process stage and written into the corresponding partition of the three-level time series database to ensure that the data is stored in an orderly manner according to the time sequence, which is convenient for subsequent querying, aggregation or export by time range.

[0077] 305. Convert the machine information into a standard industrial protocol format and report it to the upper control system so that the upper control system can generate and issue control commands based on the machine information;

[0078] In this embodiment, the conversion process maps the extracted machine information (parameters, status, alarms) into the data structure and semantics defined by standard industrial protocols (such as S6F11 messages of SECS / GEM and node data of OPC UA). The converted message is sent to the upper control system through TCP / IP or serial communication port. The upper control system parses the message content, performs logical judgment and decision-making according to the preset process recipe or real-time scheduling strategy, and generates corresponding control instructions (such as modifying parameter settings or triggering process step jumps).

[0079] 306. Receive control commands from the upper control system and parse the control commands;

[0080] In this embodiment, the system listens for and receives control commands issued by the upper control system through the communication interface, and parses the control commands. The parsing process includes: verifying the message format and integrity; extracting the command type (such as setting parameters, starting and stopping equipment, confirming alarms) and specific parameter values; converting the command content into an internally executable command object, and mapping it to the specific operation intention of the non-standard machine (such as adjusting the temperature setpoint to 80℃).

[0081] 307. Generate the corresponding keyboard and mouse operation sequence based on the parsed control instructions;

[0082] In this embodiment, the generation of operation sequence depends on a predefined instruction and operation mapping table and interface layout information. For example, for the instruction to set temperature parameters, the mapping table indicates that the following should be executed in sequence: move the mouse to the temperature input box, click to activate, enter the value on the keyboard, and press Enter to confirm. The system calculates the mouse movement path based on the current screen layout coordinates and generates a specific keyboard and mouse event sequence (including coordinates, click type, key code, delay, etc.) in combination with the instruction parameters. This sequence will be sent to the KVM hardware device for execution.

[0083] In this embodiment of the invention, by constructing a three-level time-series database (process stage, equipment number, and time dimension), the orderly storage and efficient retrieval of machine parameters, status, and alarm information are achieved. OCR recognition results are bound to timestamps to ensure data traceability. Machine information is converted into a standard industrial protocol format and reported to the upper-level control system. Control commands are received, parsed, and corresponding keyboard and mouse operation sequences are generated. This opens up data and control channels between non-standard machines and the upper-level information system. This mechanism enables non-standard equipment to seamlessly integrate into the modern CIM architecture, providing crucial data support and automated execution capabilities for intelligent manufacturing.

[0084] Please see Figure 4 In the fourth embodiment of the method for automating non-standard machine operations in this invention, steps 105 and 105 thereafter include:

[0085] 401. Execute keyboard and mouse operation sequences using KVM hardware devices, and then capture the second screen image of the non-standard machine again after executing the control.

[0086] In this embodiment, after receiving the operation sequence, the KVM hardware device sequentially performs operations such as mouse movement, clicking, and keyboard input through its simulated input interface (such as USB HID) to simulate the manual operation process. After the operation is completed, the system waits for a preset response time (such as 2 seconds). After the interface of the waiting station is refreshed and stabilized, the current screen image is captured again as the second screen image to verify the effect of the control operation.

[0087] 402. Compare the second screen image with the first screen image to obtain comparison information;

[0088] In this embodiment, the second screen image is compared with the first screen image. The comparison process uses a combination of image difference and region matching. First, the two images are aligned and corrected to eliminate slight displacement. Then, pixel-level difference images are calculated to highlight the areas where changes have occurred. At the same time, local template matching or OCR recognition results are compared in the key parameter display area (based on layout information) to quantify the degree of change. The comparison information includes the coordinates of the changed area, the type of change (such as numerical update, status icon switching, new alarm pop-up), and confidence level.

[0089] 403. Analyze the status changes of non-standard machines based on the comparison information and generate status change information;

[0090] In this embodiment, the state changes of non-standard machines are analyzed based on comparison information. The analysis process is based on the comparison information and the preset state transition logic. For example, if the value of the parameter input box is detected to change to the instruction setting value and the alarm information disappears, it is determined that the parameter setting is successful and the alarm is cleared. If there is no change in the key area or an unexpected alarm occurs, it is determined that the operation is ineffective or a new abnormality is triggered. The state change information is generated in a structured form, including change description, change level (such as success, failure, abnormality) and related screenshot evidence.

[0091] 404. Determine whether non-standard machines have reached the expected state based on status change information;

[0092] In this embodiment, the expected state is defined by the target of the control command (such as setting the temperature to 80°C or starting the device to the running state). The system will logically match the state change information with the expected state. If the change information indicates that the target has been achieved (such as matching parameter values ​​or correct state indication), it is determined that the expected state has been achieved. Otherwise, it is determined that it has not been achieved. The judgment result will trigger subsequent process branches (such as continuing execution or retrying).

[0093] 405. If not, re-execute the keyboard and mouse operation sequence;

[0094] In this embodiment, when the expected state is not achieved, the system automatically enters the retry process. Before retrying, minor adjustments can be made, such as increasing the delay between operations, fine-tuning the click coordinates, or using an alternative operation path (such as using menu navigation instead of shortcut keys). The number of retryes and adjustment strategies are configurable.

[0095] 406. If the preset number of consecutive operations fails to achieve the expected state, the operation will automatically stop and a maintenance notification will be generated.

[0096] In this embodiment, if the operation fails to reach the expected state after a preset number of consecutive attempts, the operation will automatically stop. The preset maximum number of retries can be set to 3-5 times to avoid infinite loops. If the maximum number of retries is reached and the operation still fails, the system will automatically terminate the current control task, record a detailed log (including the sequence of each operation, screen images, and comparison results), and generate a structured maintenance notification. This maintenance notification will be sent to the equipment maintenance personnel via message queue, email, or work order system. The notification will include the equipment number, the failed operation, possible causes (such as interface abnormality or hardware failure), and suggested troubleshooting measures.

[0097] 407. The machine information corresponding to the continuously acquired first screen images is associated with the time sequence according to the timestamp, and the dynamic trend curve of the core process parameters is constructed using the time sequence analysis algorithm;

[0098] In this embodiment, the system queries historical data (timestamp and value pairs) of a specific process parameter of a certain machine from a three-level time series database according to the time range. It uses time series analysis algorithms (such as moving average and exponential smoothing) to smooth and denoise the original data. Then, it plots a dynamic trend curve with the timestamp as the horizontal axis and the parameter value as the vertical axis. This dynamic trend curve can intuitively show the change law of the parameter over time (such as rising, falling, oscillating, and steady state), providing a visual basis for process monitoring and optimization.

[0099] 408. Establish a correlation mapping model between parameters;

[0100] In this embodiment, the correlation mapping model aims to reveal the mutual influence relationship between multiple process parameters. Based on historical time series data, statistical methods (such as correlation coefficient analysis, mutual information calculation) or machine learning methods (such as Granger causality test, neural network) are used to mine the linear or nonlinear correlation between parameters. The model output can be a correlation rule (such as when the pressure P1 rises, the temperature T2 usually drops after 3 seconds) or a correlation weight matrix, which can be used for subsequent anomaly root cause inference.

[0101] 409. When an alarm message is detected, locate the predetermined time window before and after the alarm occurrence time in the time series database;

[0102] In this embodiment, once an alarm message is captured from screen recognition or data reporting, the system immediately extracts the alarm occurrence time (T0). Based on predefined analysis requirements (such as tracing the root cause), a forward time window (such as [T0-60s, T0]) and a backward time window (such as [T0, T0+30s]) are set. The system extracts historical data of all relevant process parameters within these two windows from the time series database to form a local dataset for in-depth analysis.

[0103] 410. Automatically trace the parameter change trajectory within the time window in the dynamic trend curve, use the correlation mapping model to locate the root cause of the anomaly based on the parameter change trajectory, and identify the deviation parameter;

[0104] In this embodiment, the backtracking analysis first examines the trend curves of each parameter within the time window to identify the starting point and pattern of abnormal fluctuations. Combined with the correlation mapping model, the system infers the path of abnormal propagation. For example, if the model indicates that an abnormal parameter A will cause an abnormal parameter B after a delay, and the actual data conforms to this pattern, then parameter A is located as a potential root cause. At the same time, by comparing with the normal process range, parameters that significantly deviate from the set value or statistical regularity are identified and marked as deviation parameters.

[0105] 411. Generate a time-series tracing report based on the alarm occurrence time, dynamic trend curve, anomaly root cause, and deviation parameters;

[0106] In this embodiment, the time-series tracing report is a comprehensive analysis document. The time-series tracing report automatically integrates alarm details, trend curves of relevant parameters within the time window, the anomaly root cause chain (which may be multi-level) inferred based on the correlation model, a list of all identified deviation parameters and their degree of deviation. The time-series tracing report is presented in a graphic and textual format (HTML / PDF), aiming to provide process engineers with a clear timeline of abnormal events and root cause hypotheses to assist in rapid decision-making and handling.

[0107] 412. Determine the parameter type of the deviation parameter, dynamically adjust the image enhancement algorithm parameters corresponding to the parameter type, and optimize the OCR recognition model using the image enhancement algorithm parameters;

[0108] In this embodiment, the system adjusts the enhancement strategy for the display area of ​​the parameter in the image preprocessing stage according to the type of the deviation parameter (such as numerical, status, or code). For example, for small values ​​that are frequently misidentified, the intensity of local contrast enhancement can be increased in a targeted manner; for status icons, the sensitivity of edge detection can be strengthened. These adjustments form a new set of image enhancement algorithm parameters. The system uses the new parameter set to enhance the images acquired subsequently, and can add such difficult samples and their optimized enhanced images to the incremental training set of the OCR model to continuously optimize the model's recognition robustness and accuracy in such scenarios.

[0109] In this embodiment of the invention, the screen image is re-acquired and compared after the operation is performed to achieve closed-loop verification of the control effect; a retry mechanism and maintenance notification are introduced to ensure operational reliability; further, historical data in the time series database is used to construct dynamic trend curves of core process parameters and establish a correlation mapping model between parameters. When an alarm occurs, the root cause of the anomaly is automatically traced back and the deviation parameter is identified, generating a time series traceability report. At the same time, the image enhancement algorithm parameters are dynamically adjusted according to the type of deviation parameter, and the OCR recognition model is optimized, forming a continuously self-optimizing intelligent operation and maintenance system, which greatly improves the stability of non-standard machine operation and the efficiency of fault diagnosis.

[0110] The above describes the method for automating non-standard machine operations in the embodiments of the present invention. The following describes the device for automating non-standard machine operations in the embodiments of the present invention. Please refer to [link / reference]. Figure 5 One embodiment of the non-standard machine tool automation device in this invention includes:

[0111] The connection acquisition module 501 is used to acquire the interface layout information and protocol communication port information of the non-standard machine. Based on the protocol communication port information, the KVM hardware device is connected to the non-standard machine, and based on the interface layout information, the KVM hardware device is used to periodically acquire the first screen image of the non-standard machine.

[0112] The preprocessing module 502 is used to perform image preprocessing on the first screen image to obtain a preprocessed image;

[0113] The recognition and extraction module 503 is used to perform visual recognition on the preprocessed image using a preset OCR recognition model to obtain the recognition result, and extract the machine information based on the recognition result. The machine information is one or more of parameter value information, operating status information and alarm information.

[0114] The conversion and reporting generation module 504 is used to convert machine information into a standard industrial protocol format and report it to the upper control system, so that the upper control system can generate and issue control commands based on the machine information, and generate corresponding keyboard and mouse operation sequences based on the control commands.

[0115] The acquisition, analysis and generation module 505 is used to execute keyboard and mouse operation sequences using KVM hardware devices, acquire the second screen image of the non-standard machine again after execution control, analyze the status changes of the non-standard machine based on the second screen image, and generate status change information.

[0116] In this embodiment, the first screen image of the non-standard machine is periodically acquired using KVM hardware. The first screen image is preprocessed, and a preset OCR recognition model is used to perform visual recognition on the preprocessed image. Machine information is extracted based on the recognition results, converted into a standard industrial protocol format, and reported to the upper control system. The upper control system then generates and issues control commands based on the machine information. The corresponding keyboard and mouse operation sequence is generated based on the control commands, and the keyboard and mouse operation sequence is executed using KVM hardware. After the control is executed, the second screen image of the non-standard machine is acquired again. The state changes of the non-standard machine are analyzed based on the second screen image, and state change information is generated. This method is highly compatible, easy to deploy, and does not require deep modification of the non-standard equipment.

[0117] Please see Figure 6 Another embodiment of the non-standard machine tool automation device in this invention includes:

[0118] The connection acquisition module 501 is used to acquire the interface layout information and protocol communication port information of the non-standard machine. Based on the protocol communication port information, the KVM hardware device is connected to the non-standard machine, and based on the interface layout information, the KVM hardware device is used to periodically acquire the first screen image of the non-standard machine.

[0119] The preprocessing module 502 is used to perform image preprocessing on the first screen image to obtain a preprocessed image;

[0120] The recognition and extraction module 503 is used to perform visual recognition on the preprocessed image using a preset OCR recognition model to obtain the recognition result, and extract the machine information based on the recognition result. The machine information is one or more of parameter value information, operating status information and alarm information.

[0121] The conversion and reporting generation module 504 is used to convert machine information into a standard industrial protocol format and report it to the upper control system, so that the upper control system can generate and issue control commands based on the machine information, and generate corresponding keyboard and mouse operation sequences based on the control commands.

[0122] The acquisition, analysis and generation module 505 is used to execute keyboard and mouse operation sequences using KVM hardware devices, acquire the second screen image of the non-standard machine again after the control is executed, analyze the status changes of the non-standard machine based on the second screen image, and generate status change information.

[0123] In this embodiment, the acquisition and connection module 501 includes: an acquisition unit 5011, used to acquire the interface layout information and protocol communication port information of the non-standard machine; a connection unit 5012, used to connect the KVM hardware device to the non-standard machine based on the protocol communication port information; a setup unit 5013, used to set up a screenshot triggering mechanism, which includes active periodic triggering and passive alarm triggering; and an acquisition unit 5014, used to acquire the first screen image of the non-standard machine using the KVM hardware device based on the screenshot triggering mechanism and the interface layout information.

[0124] In this embodiment, the preprocessing module 502 includes: a conversion and normalization unit 5021, used to convert the first screen image into a grayscale image and perform size normalization processing on the grayscale image; a filtering, segmentation, extraction, and calculation unit 5022, used to perform median filtering on the grayscale image to suppress small reflective noise, segment the reflective area of ​​the grayscale image based on a preset grayscale threshold, extract the contour of the reflective area, and calculate the area information of the reflective area based on the contour; a replacement and repair unit 5023, used to replace the grayscale value of the reflective area with the neighborhood grayscale mean when the area information is less than or equal to the preset area threshold, and to repair the reflective area using an image repair function when the area information is greater than the preset area threshold; a normalization unit 5024, used to perform grayscale normalization processing on the repaired grayscale image; and a filtering, calculation, and fusion unit 5025, used to perform adaptive Gaussian filtering on the grayscale image and use a gradient operator. The image gradient of the grayscale image is calculated to obtain a gradient image. The gradient image is then fused with the grayscale image to enhance the text edge contours, resulting in an enhanced image. A binarization unit 5026 is used to calculate the global optimal segmentation threshold using an adaptive thresholding method and perform preliminary binarization processing on the enhanced image to obtain a preliminary binary image containing the parametric text. A segmentation binarization unit 5027 is used to divide the preliminary binary image into multiple sub-regions and calculate the local optimal threshold for each sub-region. When the deviation between the local optimal threshold and the global optimal segmentation threshold exceeds a preset range, the sub-region is re-binarized using the local optimal threshold to obtain the final binary image. An optimization and statistical return unit 5028 is used to perform morphological optimization processing on the final binary image to obtain a preprocessed image. The proportion of text pixels in the preprocessed image is statistically analyzed. When the proportion of text pixels is lower than a preset proportion threshold, the image is returned for adaptive Gaussian filtering processing.

[0125] In this embodiment, the identification and extraction module 503 includes: a construction unit 5031, used to construct a three-level time-series database based on process stage, equipment number, and time dimension; an identification unit 5032, used to perform visual recognition on the preprocessed image using a preset OCR recognition model to obtain the recognition result; an extraction unit 5033, used to extract machine information based on the recognition result, wherein the machine information is one or more of parameter value information, operating status information, and alarm information; and a binding and storage unit 5034, used to bind the machine information with the corresponding timestamp and store the machine information in the three-level time-series database.

[0126] In this embodiment, the conversion and reporting generation module 504 includes: a conversion and reporting unit 5041, used to convert machine information into a standard industrial protocol format and report it to the upper control system so that the upper control system can generate and issue control commands based on the machine information; a receiving and parsing unit 5042, used to receive control commands issued by the upper control system and parse the control commands; and a generation unit 5043, used to generate a corresponding keyboard and mouse operation sequence based on the parsed control commands.

[0127] In this embodiment, the execution acquisition, analysis, and generation module 505 includes: an execution acquisition unit 5051, used to execute a keyboard and mouse operation sequence using KVM hardware, and to acquire the second screen image of the non-standard machine again after execution control; a comparison unit 5052, used to compare the second screen image with the first screen image to obtain comparison information; an analysis and generation unit 5053, used to analyze the state changes of the non-standard machine based on the comparison information and generate state change information; a judgment unit 5054, used to judge whether the non-standard machine has reached the expected state based on the state change information; an execution unit 5055, used to re-execute the keyboard and mouse operation sequence if not; and a stop generation unit 5056, used to automatically stop the execution operation and generate a maintenance notification when the expected state is not reached after a preset number of consecutive executions.

[0128] In this embodiment, the system further includes: an association construction module 506, used to perform time-series association of machine information corresponding to the continuously acquired first screen images according to timestamps, and to construct dynamic trend curves of core process parameters using time-series analysis algorithms; an establishment module 507, used to establish an association mapping model between parameters; a positioning module 508, used to locate a predetermined time window before and after the alarm occurrence time in the time-series database when an alarm information is detected; a backtracking positioning and identification module 509, used to automatically backtrack the parameter change trajectory in the dynamic trend curve within the time window, use the association mapping model to locate the root cause of the anomaly based on the parameter change trajectory, and identify the deviation parameters; a generation module 510, used to generate a time-series traceability report based on the alarm occurrence time, dynamic trend curve, root cause of the anomaly, and deviation parameters; and a determination, adjustment, and optimization module 511, used to determine the parameter type of the deviation parameters, dynamically adjust the image enhancement algorithm parameters corresponding to the parameter type, and optimize the OCR recognition model using the image enhancement algorithm parameters.

[0129] above Figure 5 and Figure 6 The automated operation device for non-standard machines in this embodiment of the invention is described in detail from the perspective of modular functional entities. The automated operation equipment for non-standard machines in this embodiment of the invention is described in detail from the perspective of hardware processing.

[0130] Figure 7This is a schematic diagram of a non-standard machine tool automation equipment 600 provided by an embodiment of the present invention. This non-standard machine tool automation equipment 600 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 610 (e.g., one or more processors) and a memory 620, and one or more storage media 630 (e.g., one or more mass storage devices) storing application programs 633 or data 632. The memory 620 and storage media 630 can be temporary or persistent storage. The program stored in the storage media 630 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the non-standard machine tool automation equipment 600. Furthermore, the processor 610 may be configured to communicate with the storage media 630 and execute the series of instruction operations in the storage media 630 on the non-standard machine tool automation equipment 600 to implement the steps of the non-standard machine tool automation operation method provided in the above-described method embodiments.

[0131] The non-standard machine tool automation equipment 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input / output interfaces 660, and / or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 7 The illustrated structure of the equipment for automating non-standard machine operations does not constitute a limitation on the equipment for automating non-standard machine operations. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0132] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform steps to implement a method for automating non-standard machine operations.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0134] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automating the operation of non-standard machine tools, characterized in that, include: Obtain the interface layout information and protocol communication port information of the non-standard machine; connect the KVM hardware device to the non-standard machine based on the protocol communication port information; and periodically collect the first screen image of the non-standard machine based on the interface layout information using the KVM hardware device. The first screen image is preprocessed to obtain a preprocessed image; The preprocessed image is visually recognized using a preset OCR recognition model to obtain recognition results. Machine information is extracted based on the recognition results. The machine information includes one or more of parameter value information, operating status information, and alarm information. The machine information is converted into a standard industrial protocol format and reported to the upper control system, so that the upper control system can generate and issue control commands based on the machine information, and generate corresponding keyboard and mouse operation sequences based on the control commands. The keyboard and mouse operation sequence is executed using the KVM hardware device. After the control is executed, the second screen image of the non-standard machine is captured again. The state changes of the non-standard machine are analyzed based on the second screen image, and state change information is generated. The process involves using the KVM hardware device to execute the keyboard and mouse operation sequence, and after executing the control, acquiring the second screen image of the non-standard machine again. The process then analyzes the state changes of the non-standard machine based on the second screen image and generates state change information, including: The keyboard and mouse operation sequence is executed using the KVM hardware device, and the second screen image of the non-standard machine is captured again after the control is executed; The second screen image is compared with the first screen image to obtain comparison information; The status changes of the non-standard machine are analyzed based on the comparison information, and status change information is generated. Based on the status change information, determine whether the non-standard machine has reached the expected state; If not, then re-execute the keyboard and mouse operation sequence; If the preset number of consecutive operations fails to reach the expected state, the operation will automatically stop and a maintenance notification will be generated. The process of executing the keyboard and mouse operation sequence using the KVM hardware device, acquiring the second screen image of the non-standard machine again after control execution, analyzing the state changes of the non-standard machine based on the second screen image, and generating state change information further includes: The machine information corresponding to the continuously acquired first screen images is associated with the time sequence according to the timestamp, and a time sequence analysis algorithm is used to construct a dynamic trend curve of the core process parameters. Establish a mapping model between parameters; When the alarm information is detected, a predetermined time window before and after the alarm occurrence time is located in the time series database; The system automatically traces the parameter change trajectory within the time window in the dynamic trend curve, uses the correlation mapping model to locate the root cause of the anomaly based on the parameter change trajectory, and identifies the deviation parameter. A time-series tracing report is generated based on the alarm occurrence time, the dynamic trend curve, the root cause of the anomaly, and the deviation parameter; The parameter type of the deviation parameter is determined, the image enhancement algorithm parameters corresponding to the parameter type are dynamically adjusted, and the OCR recognition model is optimized using the image enhancement algorithm parameters.

2. The method for automating non-standard machine operations according to claim 1, characterized in that, The process of acquiring the interface layout information and protocol communication port information of the non-standard server, connecting the KVM hardware device to the non-standard server based on the protocol communication port information, and periodically acquiring the first screen image of the non-standard server using the KVM hardware device based on the interface layout information includes: Obtain the interface layout information and protocol communication port information of non-standard machines; Based on the protocol communication port information, the KVM hardware device is connected to the non-standard machine. Establish a screenshot triggering mechanism, which includes active periodic triggering and passive alarm triggering; Based on the screenshot triggering mechanism and the interface layout information, the first screen image of the non-standard machine is captured using the KVM hardware device.

3. The method for automating non-standard machine operations according to claim 1, characterized in that, The step of performing image preprocessing on the first screen image to obtain a preprocessed image includes: The first screen image is converted into a grayscale image, and the grayscale image is then normalized in size. The grayscale image is subjected to median filtering to suppress small reflective noise. The reflective area of ​​the grayscale image is segmented based on a preset grayscale threshold, the contour of the reflective area is extracted, and the area information of the reflective area is calculated based on the contour. When the area information is less than or equal to a preset area threshold, the gray value of the reflective area is replaced by the average gray value of the neighborhood; when the area information is greater than the preset area threshold, the reflective area is repaired using an image restoration function. The repaired grayscale image is then subjected to grayscale normalization. An adaptive Gaussian filter is applied to the grayscale image, and the image gradient of the grayscale image is calculated using a gradient operator to obtain a gradient image. The gradient image is then fused with the grayscale image to enhance the text edge contours, resulting in an enhanced image. An adaptive thresholding method is used to calculate the global optimal segmentation threshold and perform preliminary binarization processing on the enhanced image to obtain a preliminary binary image containing the parametric text. The initial binary image is divided into multiple sub-regions, and the local optimal threshold of each sub-region is calculated. When the deviation between the local optimal threshold and the global optimal segmentation threshold exceeds a preset range, the sub-region is re-binarized using the local optimal threshold to obtain the final binary image. The final binary image is subjected to morphological optimization processing to obtain a preprocessed image. The proportion of text pixels in the preprocessed image is counted. When the proportion of text pixels is lower than a preset proportion threshold, the image is returned to perform adaptive Gaussian filtering processing.

4. The method for automating non-standard machine operations according to claim 1, characterized in that, The preprocessed image is visually recognized using a preset OCR recognition model to obtain a recognition result. Machine information is then extracted based on the recognition result. This machine information includes one or more of the following: parameter value information, operating status information, and alarm information. A three-level time-series database is constructed based on process stage, equipment number, and time dimension; The preprocessed image is visually recognized using a preset OCR recognition model to obtain the recognition result; Based on the identification results, machine information is extracted, wherein the machine information is one or more of parameter value information, operating status information, and alarm information; The machine information is bound to the corresponding timestamp, and the machine information is stored in the three-level time series database.

5. The method for automating non-standard machine operations according to claim 1, characterized in that, The process of converting the machine information into a standard industrial protocol format and reporting it to the upper control system enables the upper control system to generate and issue control commands based on the machine information, and to generate corresponding keyboard and mouse operation sequences based on the control commands, including: The machine information is converted into a standard industrial protocol format and reported to the upper control system, so that the upper control system can generate and issue control commands based on the machine information. Receive the control command issued by the upper control system and parse the control command; Generate the corresponding keyboard and mouse operation sequence based on the parsed control instructions.

6. An apparatus for realizing the automated operation method of non-standard machine tools based on claim 1, characterized in that, include: The connection acquisition module is used to acquire the interface layout information and protocol communication port information of the non-standard machine. Based on the protocol communication port information, the KVM hardware device is connected to the non-standard machine, and based on the interface layout information, the KVM hardware device is used to periodically acquire the first screen image of the non-standard machine. The preprocessing module is used to perform image preprocessing on the first screen image to obtain a preprocessed image; The recognition and extraction module is used to perform visual recognition on the preprocessed image using a preset OCR recognition model to obtain recognition results, and extract machine information based on the recognition results. The machine information is one or more of parameter value information, operating status information, and alarm information. The conversion and reporting generation module is used to convert the machine information into a standard industrial protocol format and report it to the upper control system, so that the upper control system can generate and issue control commands based on the machine information, and generate corresponding keyboard and mouse operation sequences based on the control commands. The execution acquisition, analysis, and generation module is used to execute the keyboard and mouse operation sequence using the KVM hardware device, and after executing the control, to acquire the second screen image of the non-standard machine again, analyze the state changes of the non-standard machine based on the second screen image, and generate state change information.

7. A device for automating non-standard machine operations, characterized in that, The equipment for automating non-standard machine operations includes: a memory and at least one processor, wherein the memory stores instructions; At least one of the processors invokes the instructions in the memory to cause the automated non-standard machine tool operation equipment to perform the steps of the automated non-standard machine tool operation method as described in any one of claims 1-5.

8. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the method for automating non-standard machine operations as described in any one of claims 1-5.

Images