Operation and maintenance method, system and device based on image recognition, equipment and medium

Abstract

The invention relates to the technical field of image recognition, and discloses an operation and maintenance method, system and device based on image recognition, equipment and a medium. Firstly, interface image data of a target operation and maintenance system is collected, manual intervention is reduced, and especially when frequent operation is needed or a large number of systems need to be monitored, efficiency can be improved. Next, interface image data are identified and analyzed, the operation state of the target operation and maintenance system can be accurately understood, interface key data are extracted, and a basis is provided for subsequent operation and maintenance operation. Then, matching is carried out according to the interface key data and a predefined rule, and a to-be-operated instruction is generated; and finally, performing input operation simulation based on the to-be-operated instruction to automatically execute the interface operation on the target operation and maintenance system, completing the operation and maintenance task, improving the operation and maintenance working efficiency, and improving the stability and reliability of the target operation and maintenance system at the same time.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an image recognition-based operation and maintenance method, system, device, equipment, and medium. Background Technology

[0002] In industrial automation systems, the Human-Machine Interface (HMI) is an indispensable and crucial component. It serves as a vital support for ensuring the efficient and stable operation of the system and as the entry point for maintenance personnel to modify and optimize process parameters. However, in practical applications, personnel turnover makes initial system settings and routine maintenance prone to errors and vulnerabilities. Furthermore, the varying levels of expertise among maintenance personnel make it difficult for the system to maintain consistently efficient and secure operation. Therefore, a new maintenance methodology is needed to ensure the stability and security of the system. Summary of the Invention

[0003] The embodiments described in this specification aim to at least partially solve one of the technical problems in the related art. To this end, the embodiments described in this specification propose an operation and maintenance method, system, apparatus, device, and medium based on image recognition.

[0004] This specification provides an image recognition-based operation and maintenance method, the method comprising:

[0005] Collect interface image data of the target operation and maintenance system;

[0006] The interface image data is identified and parsed to extract key interface data;

[0007] Based on the key data of the interface and predefined rules, a matching process is performed to generate an operation instruction;

[0008] Based on the instruction to be operated, input operation simulation is performed to automatically execute interface operations on the target operation and maintenance system and complete the operation and maintenance task.

[0009] In one implementation, the acquisition of interface image data of the target operation and maintenance system includes:

[0010] Input operation simulation is performed based on preset task inspection and scheduling to automatically execute interface operations on the target operation and maintenance system and collect interface image data of the target operation and maintenance system.

[0011] In one implementation, the step of identifying and parsing the interface image data to extract key interface data includes:

[0012] The interface image data is preprocessed to obtain key interface elements;

[0013] Based on the key interface elements, classification and semantic analysis are performed to obtain the key interface data.

[0014] In one implementation, the step of matching the interface key data with predefined rules to generate an operation instruction includes:

[0015] The system matches key interface data from multiple target operation and maintenance systems with predefined inter-system association rules to generate cross-system operation instructions.

[0016] In one implementation, the step of matching the interface key data with predefined rules to generate an operation instruction includes:

[0017] The system matches the key data on the interface with predefined self-control rules to generate instructions for the self-control tasks to be performed.

[0018] In one embodiment, the method further includes:

[0019] An alarm mechanism is triggered based on the key data of the interface and the key data of the historical interface.

[0020] In one implementation, the step of simulating input operations based on the instruction to be operated to automatically execute interface operations on the target operation and maintenance system and complete the operation and maintenance task includes:

[0021] Actions are decomposed and sorted based on cross-system pending operation instructions and pending automatic control task instructions to generate sequence operation instructions;

[0022] The system uses a USB connection port to simulate mouse and keyboard operations based on the sequence of operation commands, thereby automating the interface operations of the target maintenance system and completing maintenance tasks.

[0023] This specification provides an image recognition-based operation and maintenance system, the device comprising:

[0024] Target operation and maintenance system;

[0025] The maintenance equipment connects to the target maintenance system via a display acquisition port and a USB connection port. The maintenance equipment acquires interface image data of the target maintenance system through the display acquisition port, identifies and parses the interface image data, and extracts key interface data. Based on the key interface data and predefined rules, the maintenance equipment generates an operation command and sends it to the target maintenance system via the USB connection port for input operation simulation, thereby automating the interface operation of the target maintenance system and completing the maintenance task.

[0026] This specification provides an image recognition-based operation and maintenance device, the device comprising:

[0027] The interface image data acquisition module is used to acquire interface image data of the target operation and maintenance system;

[0028] The image data recognition and parsing module is used to recognize and parse the interface image data and extract key interface data;

[0029] The interface key data matching module is used to match the interface key data with predefined rules and generate operation instructions.

[0030] The interface operation and system maintenance module is used to simulate input operations based on the operation command to be operated, so as to automatically execute interface operations on the target maintenance system and complete maintenance tasks.

[0031] This specification provides a computer device, a memory, and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above embodiments.

[0032] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0033] This specification provides a computer program product that includes instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0034] In the above-described implementation method, firstly, interface image data of the target operation and maintenance system is collected to reduce manual intervention, especially when frequent operations or monitoring of a large number of systems are required, thereby improving efficiency. Next, the interface image data is identified and parsed to accurately understand the operating status of the target operation and maintenance system, extracting key interface data to provide a basis for subsequent operation and maintenance operations. Then, matching is performed based on the key interface data and predefined rules to generate operation instructions. Finally, input operation simulation is performed based on the operation instructions to automatically execute interface operations on the target operation and maintenance system, completing the operation and maintenance task, improving operation and maintenance efficiency, and simultaneously enhancing the stability and reliability of the target operation and maintenance system. Attached Figure Description

[0035] Figure 1 A flowchart of operation and maintenance based on image recognition provided for the implementation of this specification;

[0036] Figure 2A flowchart illustrating the process of obtaining key interface data for the implementation of this specification;

[0037] Figure 3 A flowchart illustrating the process of completing maintenance tasks as provided in the embodiments of this specification;

[0038] Figure 4a A connection diagram of the display interface scheme provided in the embodiments of this specification;

[0039] Figure 4b A connection diagram of the camera solution provided in the embodiments of this specification;

[0040] Figure 5a A schematic diagram of the interface of the air conditioning monitoring system provided for the embodiments of this specification;

[0041] Figure 5b A schematic diagram illustrating the connection method between the maintenance equipment and the target maintenance system provided for the embodiments of this specification;

[0042] Figure 6 A schematic diagram of an image recognition-based operation and maintenance device provided for the implementation of this specification;

[0043] Figure 7 A schematic diagram of the internal structure of the computer device provided in the embodiments of the instruction manual. Detailed Implementation

[0044] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0045] In industrial automation systems, human-machine interfaces (HMIs) are an indispensable component, playing a crucial role in ensuring efficient system operation and adjusting process parameters. However, in practical engineering applications, personnel turnover leads to oversights in system configuration and maintenance. The quality of maintenance personnel training directly affects the system's efficient and safe operation. However, a large number of maintenance personnel lack sufficient professional knowledge, resulting in a shortage of qualified personnel and impacting system stability and security. Furthermore, frequent disconnections or technical staff losses at automation control system suppliers prevent timely system updates and upgrades, further exacerbating this problem.

[0046] Based on this, the embodiments in this specification provide an image recognition-based operation and maintenance method. First, interface image data of the target operation and maintenance system is collected to reduce manual intervention, especially when frequent operations or monitoring of a large number of systems are required, thus improving efficiency. Next, the interface image data is recognized and parsed to accurately understand the operating status of the target operation and maintenance system, extracting key interface data to provide a basis for subsequent operation and maintenance operations. Then, matching is performed based on the key interface data and predefined rules to generate operation instructions. Finally, input operation simulation is performed based on the operation instructions to automatically execute interface operations on the target operation and maintenance system, completing the operation and maintenance task, improving operation and maintenance efficiency, and simultaneously enhancing the stability and reliability of the target operation and maintenance system.

[0047] This specification provides an image recognition-based operation and maintenance method. Please refer to [link / reference]. Figure 1 The method may include the following steps:

[0048] S110. Collect interface image data of the target operation and maintenance system.

[0049] S120. Identify and analyze the interface image data to extract key interface data.

[0050] S130. Match the key data on the interface with predefined rules to generate the operation instructions.

[0051] S140. Simulate input operations based on the command to be operated to automatically execute interface operations on the target operation and maintenance system and complete the operation and maintenance task.

[0052] Key interface data can be text (such as numerical values ​​or error codes) or visual information (such as button colors) that can trigger maintenance operations within the interface image data. Predefined rules can be based on the experience of maintenance experts and system development, covering possible interface states, operational scenarios, and response strategies.

[0053] Specifically, the target maintenance system's interface is captured via screen capture or video stream acquisition using image acquisition devices or display interfaces, without relying on the original manufacturer's communication protocols or data point tables. This allows for real-time acquisition of the target maintenance system's human-computer interaction interface images or video streams, thus obtaining the target maintenance system's interface image data. It should be noted that the interface image data can take various forms; it can be static images, dynamic videos, or a combination of images and videos to meet diverse needs in different maintenance scenarios. Next, a deep learning model is used to identify and parse the acquired interface image data, extracting key interface data to provide data for subsequent maintenance. After extracting the key interface data, it is matched against predefined rules to determine the corresponding parts of the key interface data within the predefined rules, thereby generating operation instructions. These operation instructions specify the specific actions the target maintenance system needs to perform in the current interface state, such as clicking a button, entering text, or selecting a menu item. Finally, the operation instructions are simulated to automatically execute the interface operations of the target maintenance system, thereby completing the maintenance task. For example, the target operation and maintenance system can be monitored 24 / 7 to meet the requirements of high real-time scenarios.

[0054] In the above implementation, firstly, interface image data of the target operation and maintenance system is collected to reduce manual intervention, especially when frequent operations or monitoring of a large number of systems are required, thereby improving efficiency. Next, the interface image data is identified and parsed to accurately understand the operating status of the target operation and maintenance system, extracting key interface data to provide a basis for subsequent operation and maintenance operations. Then, matching is performed based on the key interface data and predefined rules to generate operation instructions. Finally, input operation simulation is performed based on the operation instructions to automatically execute interface operations on the target operation and maintenance system, completing the operation and maintenance task, improving operation and maintenance efficiency, and simultaneously enhancing the stability and reliability of the target operation and maintenance system.

[0055] In some implementations, collecting interface image data of the target operation and maintenance system may include: simulating input operations based on a preset task inspection schedule to automatically perform interface operations on the target operation and maintenance system and collect interface image data of the target operation and maintenance system.

[0056] Specifically, in actual operation and maintenance scenarios, it is common to encounter situations where the target operation and maintenance system contains multiple executable interfaces, or where multiple target operation and maintenance systems need to be maintained simultaneously. Given this, the number of interfaces to be inspected increases, and relying solely on manual inspection is not only inefficient but also prone to oversights. Therefore, it is necessary to optimize and design the inspection process. First, before commencing operation and maintenance work, based on the established operation and maintenance schedule, a detailed inspection process plan can be developed for each interface and different operation and maintenance systems to be included in the inspection scope, thereby generating a preset task inspection schedule. After determining the preset task inspection schedule, based on the schedule, relevant operation and maintenance tools and scripting technologies are used to automatically generate operation and maintenance task scripts. These scripts can include various operation and maintenance instructions, such as operation steps. Input operation simulation is performed based on the task scripts to generate corresponding mouse and keyboard actions, ensuring that the scripts can accurately interact with the target operation and maintenance system interfaces. Next, following the simulated operation steps, the operation processes for each interface of the target operation and maintenance system are automatically executed. During execution, the current interface of the target operation and maintenance system will be collected simultaneously to obtain real-time interface image data. By analyzing this interface image data, it is possible to determine in real time and intuitively whether there are any areas in the inspection process that need adjustment or modification, thereby enabling necessary operation and maintenance operations to ensure the stable and efficient operation of the operation and maintenance system.

[0057] In the above embodiments, input operation simulation is performed based on preset task inspection arrangement to automatically execute interface operations on the target operation and maintenance system and collect interface image data of the target operation and maintenance system, thereby realizing automated inspection without the need for manual interface switching operations and improving operation and maintenance efficiency.

[0058] In some implementations, please refer to Figure 2 The process of identifying and parsing interface image data to extract key interface data may include the following steps:

[0059] S210. Perform image preprocessing on the interface image data to obtain key interface elements.

[0060] S220. Classify and semantically analyze key interface elements to obtain key interface data.

[0061] Specifically, after acquiring the interface image data, image preprocessing is performed. Preprocessing includes denoising to eliminate useless information and interference in the image; image enhancement to improve image clarity and contrast; and image segmentation to divide the image into multiple meaningful regions. After obtaining key interface elements through image preprocessing, deep learning models (such as Convolutional Neural Networks (CNNs) and Transformer models) can be used to classify and semantically analyze these elements to extract data (such as status indicators, numerical displays, error messages, etc.) and obtain key interface data. Finally, the identified key interface data is stored in a real-time database for immediate monitoring and rapid response. Simultaneously, this key data is also stored in a historical database for long-term data analysis and trend analysis. Key interface elements can include text, icons, controls, etc.

[0062] In the above implementation, the interface image data is preprocessed to obtain key interface elements. Based on the key interface elements, classification and semantic parsing are performed to obtain key interface data, providing a data foundation for subsequent system operation and maintenance.

[0063] In some implementations, matching key interface data with predefined rules to generate operation instructions may include: matching key interface data of multiple target operation and maintenance systems with predefined inter-system association rules to generate cross-system operation instructions.

[0064] Among them, the predefined inter-system association rules can be formulated based on long-term operation and maintenance experience, system design principles, and the overall operating logic of the building or equipment. The predefined inter-system association rules can accurately describe the interaction patterns and linkage mechanisms between different operation and maintenance systems in various scenarios.

[0065] Specifically, because there are interrelationships between operation and maintenance systems—for example, common lighting control systems and air conditioning control systems—although they operate independently, they influence and constrain each other in terms of overall building energy management and operational efficiency optimization. Therefore, to achieve comprehensive and accurate operation and maintenance, it is necessary to acquire key interface data from multiple target operation and maintenance systems to reflect their current operational status. Then, the key interface data from multiple target operation and maintenance systems is matched with predefined inter-system association rules to uncover potential operational patterns, revealing important information such as cross-system collaborative operation issues, potential fault chain reaction paths, and key nodes in overall operational efficiency, generating cross-system pending operation instructions. These instructions specify the operation and maintenance system requiring operation in the current interface state, as well as the specific action to be performed; for example, clicking the air conditioning system's "off" button.

[0066] In the above implementation, cross-system operation instructions are generated by matching key interface data of multiple target operation and maintenance systems with predefined inter-system association rules. This method is suitable for distributed system environments, can adapt to the graphical interfaces of various software systems, does not require modification of system code, and can effectively realize joint control of multiple systems.

[0067] In some implementations, matching key interface data with predefined rules to generate an operation instruction may include: matching key interface data with predefined self-control rules to generate an operation self-control task instruction.

[0068] Among them, predefined self-control rules can be rules that users pre-define to control and manage the behavior of the target operation and maintenance system according to specific needs and goals.

[0069] Specifically, in addition to pre-set general rules, users can also be provided with flexible customization options to determine predefined self-control rules that fit actual needs. Key interface data is matched with predefined self-control rules to uncover potential operational patterns, revealing operational problems and potential faults, and generating instructions for self-control tasks to be performed. These instructions specify the concrete actions that the target maintenance system needs to perform in the current interface state, such as clicking a button, entering text, or selecting a menu item.

[0070] In some implementations, based on preset task inspections, predefined self-control rules can be further used to determine page navigation or specific actions to be performed, generating instructions for self-control tasks to be executed. For example, during preset task inspections, if key data on a certain interface is detected to exceed a preset threshold, the next page to navigate to or the text to be entered can be determined according to predefined self-control rules.

[0071] In the above implementation, key interface data and predefined self-control rules are matched to generate instructions for self-control tasks to be operated, thereby improving operation and maintenance efficiency and accuracy.

[0072] In some implementations, the method may further include: triggering an alarm mechanism based on key interface data and historical key interface data.

[0073] Specifically, key interface data is compared with pre-set thresholds to determine if any anomalies exist in the current system operation. A comprehensive trend analysis of historical interface data is performed, observing long-term patterns, fluctuations, and growth or decline trends to determine if any anomalies exist. Upon detection of anomalies, a multi-level alarm mechanism is triggered. This mechanism categorizes and handles alarms according to their severity and urgency, ensuring timely and accurate communication to relevant personnel. Alarm methods include email notifications, instant SMS alerts, or system notifications. The pre-set thresholds can be normal range limits determined based on a comprehensive consideration of historical data, industry standards, or system design requirements.

[0074] In the above implementation, an alarm mechanism is triggered based on key interface data and historical key interface data to improve system stability and security.

[0075] In some implementations, please refer to Figure 3 The process of simulating input operations based on the command to be operated, in order to automatically execute interface operations on the target operation and maintenance system and complete the operation and maintenance task, may include the following steps:

[0076] S310: Based on cross-system pending operation instructions and pending automatic control task instructions, decompose and sort the actions to generate a sequence of operation instructions.

[0077] The S320 uses a USB connection port to simulate mouse and keyboard operations based on sequence commands to automate interface operations on the target maintenance system and complete maintenance tasks.

[0078] Specifically, since cross-system monitoring and automatic control task monitoring can occur simultaneously, after receiving cross-system pending operation instructions and pending automatic control task instructions, given the complexity of the operation instructions and the priority requirements of different instructions, it is necessary to decompose these instructions into action steps. Complex instructions are broken down into several simple, clear, and executable basic operation steps to facilitate subsequent processing and execution. After decomposition, the decomposed operation actions are then rationally ordered based on factors such as the urgency of the maintenance task, the dependencies between different systems, and the logical order of operations, ultimately generating a complete and ordered sequence of operation instructions. Next, through a USB connection port, mouse and keyboard operations are precisely simulated according to the generated sequence of operation instructions to automate the interface operations of the target maintenance system, thereby improving the efficiency and accuracy of maintenance work and ensuring that maintenance tasks are completed in a timely and effective manner.

[0079] In the above implementation, actions are decomposed and sorted based on cross-system pending operation instructions and pending automatic control task instructions to generate a sequence of operation instructions. Mouse and keyboard simulation is performed based on the sequence of operation instructions through a USB connection port to automatically execute interface operations on the target operation and maintenance system, complete operation and maintenance tasks, realize multi-task collaborative processing, and enhance the accuracy of operations.

[0080] In some implementations, when under the same or similar environment and conditions, preset task inspection schedules and predefined self-control rules can be conveniently invoked to achieve efficient task processing and system operation.

[0081] This specification provides an image recognition-based operation and maintenance system, which may include:

[0082] Target operation and maintenance system.

[0083] The maintenance equipment connects to the target maintenance system via a display acquisition port and a USB connection port. The equipment acquires interface image data from the target maintenance system through the display acquisition port, identifies and parses the interface image data, and extracts key interface data. Based on the key interface data and predefined rules, the equipment generates an operation command and sends it to the target maintenance system via the USB connection port for input operation simulation. This automates the execution of interface operations on the target maintenance system, completing the maintenance task.

[0084] Specifically, the target maintenance system and the maintenance equipment are connected via a display acquisition port. The display acquisition port acquires interface image data from the target maintenance system and transmits this data to the maintenance equipment. The maintenance equipment identifies and parses the interface image data, extracts key interface data, and then matches this data with predefined rules to generate an operation command. After generating the operation command, the maintenance equipment sends it to the target maintenance system via a USB connection port for input operation simulation, thereby automating the interface operations on the target maintenance system and completing the maintenance task.

[0085] For example, please refer to Figure 4a The target maintenance system's display interface and the maintenance equipment's acquisition interface are connected via VGA / HDMI / DP to transmit interface image data. Additionally, the target maintenance system and the maintenance equipment are connected via USB to enable simulated mouse and keyboard operation.

[0086] Please see Figure 4bThe system uses a camera to capture interface image data of the target maintenance system. The camera connects to the maintenance equipment via VGA / HDMI / DP / USB / LAN to transmit interface image data. Additionally, the target maintenance system connects to the maintenance equipment via USB to enable simulated mouse and keyboard operation.

[0087] It should be noted that, in addition to simulating mouse and keyboard input to automate interface operations on the target maintenance system, the maintenance equipment also integrates a variety of practical auxiliary functions, such as data recognition, monitoring, data storage, and alarm functions.

[0088] In the above embodiments, the device connects to the target maintenance system via a display acquisition port and a USB connection port. The maintenance device acquires the interface image data of the target maintenance system through the display acquisition port, identifies and parses the interface image data, and extracts key interface data. The maintenance device matches the key interface data with predefined rules, generates an operation command, and then sends the operation command to the target maintenance system via the USB connection port for input operation simulation. This automates the execution of interface operations on the target maintenance system, completing the maintenance task. The maintenance device and the maintained system are physically isolated from the network, avoiding security risks caused by network connection.

[0089] This specification provides an application scenario for an image recognition-based operation and maintenance method, using an air conditioning monitoring system as an example. Please refer to [link / reference]. Figure 5a , Figure 5a The system's air conditioning monitoring features a graphical interface displaying key metrics such as fresh air temperature, fresh air humidity, return air temperature, valve opening, and fan frequency meter power. Please refer to [link / reference]. Figure 5b The client computer host of the air conditioning monitoring system is connected to the maintenance equipment via HDMI and USB. The maintenance equipment captures data from the monitoring interface of the air conditioning monitoring system through HDMI, obtaining interface image data. The image preprocessing module of the maintenance equipment performs grayscale processing and region segmentation on the interface image data, and the element recognition module extracts operational data to obtain key interface data. The key interface data is shown in the table below:

[0090]

[0091] The data storage module of the maintenance equipment stores key interface data in real-time and historical databases. Then, it matches the key interface data with predefined rules. For example, when the fresh air temperature is above 33 degrees Celsius and the return air CO2 concentration is below 600 ppm, the system monitors whether the opening of the fresh air valve exceeds 20%. If the fresh air valve opening does exceed 20%, the system will issue a command to modify the fresh air valve opening to 19%, and simultaneously generate corresponding action commands, namely, clicking the corresponding location on the air conditioning monitoring system via USB simulated mouse and inputting "20" via keyboard.

[0092] An alarm mechanism is triggered based on key data displayed on the interface. For example, if the return air temperature display is not in numeric form, and it is confirmed as an illegal string after three consecutive identifications, it is suspected that the return air temperature sensor is malfunctioning. In this case, maintenance personnel will be notified to repair the sensor via interface prompts, SMS, or email.

[0093] Comprehensive analysis can also be performed. For example, if the supply air temperature is set to 20 degrees Celsius, but the actual supply air temperature fails to reach the set value and exceeds 20 degrees Celsius for 30 consecutive minutes, an action command to click the "Cold Source System" menu will be automatically generated to check whether the chilled water outlet temperature of the cold source system is within the normal range. If the chilled water outlet temperature is found to be out of range, the operation and maintenance personnel will be notified for further investigation via interface prompts, SMS, or email.

[0094] This specification provides an image recognition-based operation and maintenance device 600. Please refer to [link to documentation]. Figure 6 The image recognition-based operation and maintenance device 600 includes: an interface image data acquisition module 610, an image data recognition and parsing module 620, an interface key data matching module 630, and an interface operation and system operation and maintenance module 640.

[0095] The interface image data acquisition module 610 is used to acquire interface image data of the target operation and maintenance system.

[0096] The image data recognition and parsing module 620 is used to recognize and parse the interface image data and extract key interface data;

[0097] The interface key data matching module 630 is used to match the interface key data and predefined rules to generate an operation instruction;

[0098] The interface operation and system maintenance module 640 is used to simulate input operations based on the operation command to be operated, so as to automatically execute the interface operation of the target maintenance system and complete the maintenance task.

[0099] For a detailed description of the image recognition-based operation and maintenance device, please refer to the description of the image recognition-based operation and maintenance method above, which will not be repeated here.

[0100] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image recognition-based operation and maintenance method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0101] Those skilled in the art will understand that Figure 7 The structures shown are merely block diagrams of some structures related to the solutions disclosed in this specification, and do not constitute a limitation on the computer device to which the solutions disclosed in this specification are applied. Specifically, the computer device may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0102] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps described above.

[0103] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0104] One embodiment of this specification provides a computer program product including instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.

[0105] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

Claims

1. An operation and maintenance method based on image recognition, characterized in that, The method includes: Collect interface image data of the target operation and maintenance system; The interface image data is identified and parsed to extract key interface data; Based on the key data of the interface and predefined rules, a matching process is performed to generate an operation instruction; Based on the instruction to be operated, input operation simulation is performed to automatically execute interface operations on the target operation and maintenance system and complete the operation and maintenance task.

2. The method according to claim 1, characterized in that, The interface image data of the target operation and maintenance system to be collected includes: Input operation simulation is performed based on preset task inspection and scheduling to automatically execute interface operations on the target operation and maintenance system and collect interface image data of the target operation and maintenance system.

3. The method according to claim 1, characterized in that, The process of identifying and parsing the interface image data to extract key interface data includes: The interface image data is preprocessed to obtain key interface elements; Based on the key interface elements, classification and semantic analysis are performed to obtain the key interface data.

4. The method according to claim 1, characterized in that, The step of matching the key interface data and predefined rules to generate an operation instruction includes: The system matches key interface data from multiple target operation and maintenance systems with predefined inter-system association rules to generate cross-system operation instructions.

5. The method according to claim 1 or 2, characterized in that, The step of matching the key interface data and predefined rules to generate an operation instruction includes: The system matches the key data on the interface with predefined self-control rules to generate instructions for the self-control tasks to be performed.

6. The method according to claim 1, characterized in that, The method further includes: An alarm mechanism is triggered based on the key data of the interface and the key data of the historical interface.

7. The method according to claim 1, characterized in that, The step of simulating input operations based on the instruction to be operated, to automatically execute interface operations on the target operation and maintenance system and complete the operation and maintenance tasks, includes: Actions are decomposed and sorted based on cross-system pending operation instructions and pending automatic control task instructions to generate sequence operation instructions; The system uses a USB connection port to simulate mouse and keyboard operations based on the sequence of operation commands, thereby automating the interface operations of the target maintenance system and completing maintenance tasks.

8. An operation and maintenance system based on image recognition, characterized in that, The system includes: Target operation and maintenance system; The maintenance equipment connects to the target maintenance system via a display acquisition port and a USB connection port. The maintenance equipment acquires interface image data of the target maintenance system through the display acquisition port, identifies and parses the interface image data, and extracts key interface data. Based on the key interface data and predefined rules, the maintenance equipment generates an operation command and sends it to the target maintenance system via the USB connection port for input operation simulation, thereby automating the interface operation of the target maintenance system and completing the maintenance task.

9. An operation and maintenance device based on image recognition, characterized in that, The device includes: The interface image data acquisition module is used to acquire interface image data of the target operation and maintenance system; The image data recognition and parsing module is used to recognize and parse the interface image data and extract key interface data; The interface key data matching module is used to match the interface key data with predefined rules and generate operation instructions. The interface operation and system maintenance module is used to simulate input operations based on the operation command to be operated, so as to automatically execute interface operations on the target maintenance system and complete maintenance tasks.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images