Intelligent operation and maintenance method, device and system based on visual algorithm and electronic equipment

Abstract

The application relates to the field of intelligent operation and maintenance control, in particular to an intelligent operation and maintenance method, device and system based on a visual algorithm and electronic equipment; the method comprises the following steps: if a first fault event is triggered again under the condition that a data completion event is triggered in advance in a digital twin model, generating an image acquisition instruction for an object device corresponding to the data completion event, collecting a live image of the object device through a target device, extracting running data of the object device from the live image through a visual algorithm, and further feeding back the running data to the digital twin model, so that the digital twin model can make fault prediction again; if the digital twin model triggers a second fault event which is the same as a fault device corresponding to the first fault event again, it is determined that the first event is valid, otherwise, the first event is invalid. Through the method, the operation and maintenance cost of the system can be reduced.

Description

Technical Field

[0001] This application relates to the field of intelligent operation and maintenance control, and in particular to an intelligent operation and maintenance method, device, system and electronic device based on vision algorithms. Background Technology

[0002] With the development of technology, digital twin technology has been widely and maturely applied in scenarios such as smart grids, smart factory systems, building monitoring systems in industrial parks, and digital transportation systems. A digital twin model includes a data mapping model and a fault prediction model. The data mapping model maps system data in real time, while the fault prediction model predicts the system's state based on real-time data provided by the data mapping model and historical system data. This enables operation and maintenance management such as fault prediction and intelligent scheduling.

[0003] Typically, the system transmits data from each device (node) to the data mapping model wirelessly. However, data loss can occur within the data mapping model due to sensor malfunctions within the devices or packet loss during data transmission. To maintain the stable operation of the digital twin model, the lost data is usually supplemented using a predictive model or a diffusion model, and then the supplemented data is sent to the fault prediction model for subsequent fault prediction.

[0004] However, these supplementary data often contain biases, which may lead to incorrect fault judgments from the fault prediction model. Since faults in power systems usually require maintenance personnel to investigate and repair, incorrect fault prediction results will greatly increase the manpower costs of system maintenance.

[0005] Therefore, how to reduce the cost of system operation and maintenance is an urgent problem to be solved. Summary of the Invention

[0006] Therefore, it is necessary to provide an intelligent operation and maintenance method, device, system, and electronic device based on vision algorithms that can reduce operation and maintenance costs, addressing the aforementioned technical problems.

[0007] Firstly, this application provides an intelligent operation and maintenance method based on visual algorithms, the method being applied to an intelligent operation and maintenance system, the method comprising:

[0008] If it is determined that the digital twin model has triggered a first fault event, determine whether a data completion event was triggered in the digital twin model during the historical time period prior to the first fault event being triggered;

[0009] If the data completion event is determined to have been triggered, the target device corresponding to the completed data is determined, and an image acquisition command for the target device is generated.

[0010] The image acquisition command is sent to the target device to obtain the on-site image acquired by the target device in response to the image acquisition command.

[0011] The on-site images are processed to extract data, and the operating data of the target device is obtained. The operating data is then sent to the digital twin model.

[0012] If it is determined that the digital twin model does not trigger the second fault event within a preset time period, or the second fault event triggered by the digital twin model corresponds to a different fault device than the first fault event, then the first fault event is determined to be invalid.

[0013] If, within a preset time period, the digital twin model is determined to trigger the same faulty device as the first fault event, then the first fault event is determined to be valid, and a troubleshooting task for the first fault event is generated.

[0014] In one embodiment, the digital twin model includes an interaction module that completes missing data in the output data of the data mapping model and transmits the completed output data to the fault prediction model; determining whether a data completion event was triggered in the digital twin model within the historical time period prior to the first fault event being triggered includes:

[0015] Send a query request to the interaction module to receive the query results obtained by the interaction module in response to the query request by querying the execution log of the data completion event;

[0016] Based on the query results, determine whether a data completion event was triggered within the digital twin model during the historical time period prior to the first fault event being triggered.

[0017] In one embodiment, generating image acquisition instructions for the target device includes:

[0018] Determine the device type and location of the target device;

[0019] An image acquisition command is generated based on the device type and the device location.

[0020] In one embodiment, the step of performing data extraction processing on the on-site image to obtain the operating data of the target device includes:

[0021] Target recognition is performed on the on-site images to obtain the image regions corresponding to each type of device;

[0022] A target image region is determined from each of the image regions, wherein the target image region is the image region corresponding to the object device or an associated device that has a signal transmission relationship with the object device;

[0023] The target image region is subjected to text recognition to obtain the operating data of the target device.

[0024] In one embodiment, the step of performing text recognition on the target image region to obtain the operating data of the target device includes:

[0025] Perform text recognition on the target image region to obtain a data form containing operational data corresponding to at least one device;

[0026] Based on the device type and data type of the object device, the corresponding operating data of the object device is determined from the data form.

[0027] In one embodiment, sending the operational data to the digital twin model includes:

[0028] The operational data is converted to obtain standard data that is compatible with the input format of the fault prediction model.

[0029] The standard data and the data stream sent from the data mapping model to the fault prediction model are time-aligned, and the aligned standard data is then sent to the digital twin model.

[0030] In one embodiment, the method further includes determining that the first fault event is valid and generating a fault investigation task for the first fault event if it is determined that the data completion event was not triggered in the digital twin model during the historical time period prior to the first fault event being triggered.

[0031] Secondly, this application also provides an intelligent operation and maintenance device based on visual algorithms. The device includes an event determination module, an instruction generation module, an image acquisition module, a data extraction module, and a validity judgment module, wherein:

[0032] The event determination module is used to determine whether a data completion event was triggered in the digital twin model during the historical time period before the first fault event was triggered, in the case where the digital twin model is determined to have triggered a first fault event.

[0033] The instruction generation module is used to determine the target device corresponding to the data completion if the data completion event is determined to be triggered, and to generate an image acquisition instruction for the target device.

[0034] An image acquisition module is used to send the image acquisition command to the target device to obtain the on-site image acquired by the target device in response to the image acquisition command.

[0035] The data extraction module is used to extract and process data from the on-site images to obtain the operating data of the target device, and send the operating data to the digital twin model;

[0036] The validity determination module is used to determine that the first fault event is invalid if it is determined that the digital twin model has not triggered the second fault event within a preset time period, or if the second fault event triggered by the digital twin model corresponds to a different fault device than the first fault event.

[0037] If, within a preset time period, the digital twin model is determined to trigger the same faulty device as the first fault event, then the first fault event is determined to be valid, and a troubleshooting task for the first fault event is generated.

[0038] Thirdly, this application also provides an intelligent operation and maintenance system, which includes a target device, a subscription unit, a data interaction layer, a visual algorithm layer, and a perception layer, wherein:

[0039] The subscription unit is linked to the data interaction layer and the perception layer, and the subscription unit is used to subscribe to various times that originate within the digital twin model;

[0040] The data interaction layer interacts with the target device, and is linked to the visual algorithm layer. The data interaction layer is used to determine, in the event that the digital twin model triggers a first fault event, whether a data completion event was triggered within the digital twin model during a historical time period prior to the triggering of the first fault event; if the data completion event is determined to have been triggered, then the target device corresponding to the completed data is identified, and an image acquisition command is generated for the target device; the image acquisition command is sent to the target device to obtain the on-site image acquired by the target device in response to the image acquisition command.

[0041] The target device is used to collect on-site images of various devices within the system;

[0042] The visual algorithm layer is linked to the digital twin model and the perception layer. The visual algorithm layer is used to extract and process data from the on-site images to obtain the operating data of the object device, and then send the operating data to the digital twin model.

[0043] The perception layer is used to determine that the first fault event is invalid if it is determined that the digital twin model has not triggered the second fault event within a preset time period, or if the second fault event triggered by the digital twin model corresponds to a different fault device than the first fault event; and to determine that the first fault event is valid if it is determined that the second fault event triggered by the digital twin model corresponds to the same fault device as the first fault event within a preset time period, and to generate a fault investigation task for the first fault event.

[0044] Fourthly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the intelligent operation and maintenance method based on a vision algorithm as described in any one of the first aspects above.

[0045] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent operation and maintenance method based on a vision algorithm as described in any one of the first aspects above.

[0046] The effectiveness (accuracy) of the first fault event triggered after the data completion event has been pre-triggered by the digital twin model in the aforementioned intelligent operation and maintenance method, device, system, and electronic equipment based on visual algorithms is difficult to determine. Therefore, an image acquisition command is further generated for the target device corresponding to the data completion event. The target device acquires on-site images of the target device, and the visual algorithm extracts the operating data of the target device from the on-site images. The operating data is then fed back to the digital twin model, enabling the digital twin model to make a new fault prediction. If the digital twin model re-triggers a second fault event that is the same as the faulty device corresponding to the first fault event, then the first event is determined to be valid; otherwise, it is invalid.

[0047] By acquiring on-site images of the target equipment and extracting actual operational data using visual processing algorithms, the digital twin model can obtain accurate data to re-predict faults. The validity of the first fault event is determined by comparing the results of the two fault predictions, thereby reducing the probability of incorrect fault predictions due to incomplete data. This improves the realism and accuracy of fault events triggered by the digital twin model, reduces the waste of human and material resources caused by erroneous fault predictions, and ultimately lowers maintenance costs. At this point, a fault investigation task for the first fault event is generated to facilitate on-site investigation by maintenance personnel. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is an application environment diagram of a vision-based intelligent operation and maintenance method in one embodiment;

[0050] Figure 2 This is a schematic diagram of a vision-based intelligent operation and maintenance process in one embodiment;

[0051] Figure 3 This is a flowchart illustrating the steps for extracting runtime data in one embodiment;

[0052] Figure 4 This is a structural block diagram of a vision-based intelligent device in one embodiment;

[0053] Figure 5 This is a schematic diagram of the architecture of an intelligent operation and maintenance system in one embodiment;

[0054] Figure 6 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0057] The intelligent operation and maintenance method based on vision algorithms provided in this application can be applied to, for example... Figure 1In the application scenario shown, the intelligent operation and maintenance system S1 communicates with the digital twin model S2 of the target system via a network. The data storage system can store the data that the digital twin model S2 needs to process. The data storage system can be integrated on a server or placed in the cloud or other network servers. The digital twin model S2 includes a data mapping model S21, a fault prediction model S22, and an interaction module S23. The data mapping model maps the system's data in real time. The interaction module S23 is used to complete the missing data in the output data of the data mapping model S21. The fault prediction model S22 predicts the system's state based on the completed output data and the system's historical data, obtains the fault prediction result, and triggers a fault event.

[0058] The target system is a system containing multiple devices, such as a smart grid system, a production / monitoring system in a smart factory, and a monitoring system for smart transportation. In this application embodiment, a smart grid is used as an example to illustrate the application scenario of the vision-based intelligent operation and maintenance method; however, this does not constitute a limitation on the method provided in this application embodiment. The vision-based intelligent operation and maintenance method provided in this application embodiment can also be applied to other application scenarios, such as smart transportation management systems and smart factory monitoring systems.

[0059] The intelligent operation and maintenance system S1 is configured on an electronic device, which can be a personal computer or a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0060] In one exemplary embodiment, such as Figure 2 As shown, a visual algorithm-based intelligent operation and maintenance method is provided, which is then applied to... Figure 1 Taking the intelligent operation and maintenance system in the example, the explanation includes the following steps 110-160, wherein:

[0061] Step 110: If it is determined that the digital twin model has triggered the first fault event, determine whether a data completion event has been triggered in the digital twin model during the historical time period before the first fault event was triggered.

[0062] In this embodiment of the application, the intelligent operation and maintenance system includes a subscription module. The subscription module can subscribe to data completion events triggered by the interaction module and fault events triggered by the fault prediction module in the digital twin model. Specifically, when the digital twin model triggers any event, the digital twin module sends the type of the triggered event and other event information as a subscription message to the subscription module. Therefore, it is possible to directly determine whether the digital twin model has triggered a first fault event based on the subscription message received by the subscription module.

[0063] Furthermore, after determining that the digital twin model triggered the first fault event, the subscribed messages are further analyzed to determine whether the interactive modules within the digital twin model triggered a data completion event during the historical time period prior to the triggering of the first fault event. If it is determined that a data completion event was triggered, it indicates a possible relationship between the triggering of the first fault event and the completed data corresponding to the data completion event. The duration of the historical time period can be preset, for example, it can be 5 minutes or 10 minutes; the specific length of the historical time period is not specifically limited in this embodiment.

[0064] If it is determined that no data completion event was triggered in the digital twin model during the historical time period prior to the first fault event, then the first fault event is deemed valid, and a fault investigation task for the first fault event is generated.

[0065] Step 120: If it is determined that a data completion event has been triggered, then determine the target device corresponding to the completed data and generate an image acquisition command for the target device.

[0066] In this embodiment of the application, if it is determined that the interaction module of the digital twin model triggered a data completion event within a historical time period, the target device corresponding to the data completion event is further determined through the event information received by the subscription module; wherein, the target device and the device corresponding to the missing data in the data output by the data mapping model are also the devices corresponding to the data completed by the interaction module. Further, the device type and device location of the target device are determined; then, an image acquisition command is generated based on the device type and device location.

[0067] Step 130: Send the image acquisition command to the target device to obtain the on-site image acquired by the target device in response to the image acquisition command.

[0068] In one possible implementation of this application, image acquisition devices can be pre-positioned at multiple preset locations within the target system, each capable of acquiring images of devices within its field of view. In this case, the target device is determined from among these image acquisition devices.

[0069] The intelligent operation and maintenance system identifies the target device from among various image acquisition devices based on the location information and device type of the target device, and sends the image acquisition command to the target device.

[0070] In another possible implementation, the image acquisition device can also be an inspection device, such as an inspection vehicle or robot, that performs inspections within the target system's site; in this case, the target device is the inspection device. The intelligent operation and maintenance system sends the generated image acquisition commands to the target device.

[0071] It should be noted that, due to the large number of devices within the target system, on the one hand, the operational data collected by the devices can be wirelessly transmitted to the intelligent operation and maintenance system directly through the built-in sensors of the devices. On the other hand, for the convenience of on-site operation and maintenance, the target system is also divided into sections, with edge acquisition devices set up in each area. The edge acquisition devices collect data from each device and display the data on a screen, which greatly facilitates on-site operation and maintenance personnel to understand the real-time operating status of each device.

[0072] Furthermore, after receiving the image acquisition command, the target device acquires images based on the image acquisition command to obtain a field image containing the target device and the corresponding edge devices.

[0073] Step 140: Extract and process the data from the on-site images to obtain the operating data of the target device, and send the operating data to the digital twin model.

[0074] In this embodiment of the application, visual algorithms are used to extract and process data from on-site images to obtain the operational data of the target device, and this operational data is then sent to the interactive module or fault prediction model in the digital twin model. It should be further explained that the digital twin model processes data from each device over time.

[0075] Step 150: If it is determined that the digital twin model did not trigger the second fault event within the preset time period, or the second fault event triggered by the digital twin model corresponds to a different fault device than the first fault event, then the first fault event is determined to be invalid.

[0076] Step 160: If, within a preset time period, the digital twin model triggers the same faulty device as the first fault event, then the first fault event is determined to be valid, and a troubleshooting task for the first fault event is generated.

[0077] In this embodiment of the application, the operating data of the target device is sent to the digital twin model, and the digital twin model will re-diagnose the fault of the target system; if it is determined that the digital twin model has not triggered a second fault event within a preset time period.

[0078] If the digital twin model is found to have triggered a second fault event within a preset time period, the faulty device targeted by the second fault event is further compared with the faulty device corresponding to the first fault event. If the faulty device targeted by the second fault event is different from the faulty device targeted by the first fault event, it can be determined that the first fault event triggered by the fault prediction model due to the supplementary data of the target device is invalid, and its invalidity is due to the error of the supplementary data.

[0079] If the faulty device targeted by the second fault event is the same as the faulty device targeted by the first fault event, it can be determined that the operating data of the target device has not affected the validity of the first fault event, that is, the first fault event is valid; at this time, a fault investigation task for the first fault event is generated and sent to the corresponding maintenance equipment of the maintenance team members so that the maintenance personnel can handle the fault in a timely manner.

[0080] In the aforementioned intelligent operation and maintenance method based on visual algorithms, the validity (accuracy) of the first fault event triggered after the data completion event has been pre-triggered by the digital twin model is difficult to determine. Therefore, an image acquisition command is further generated for the target device corresponding to the data completion event. The target device acquires on-site images of the target device, and the visual algorithm extracts the operating data of the target device from the on-site images. The operating data is then fed back to the digital twin model, enabling the digital twin model to make a new fault prediction. If the digital twin model re-triggers a second fault event that is the same as the faulty device corresponding to the first fault event, then the first event is determined to be valid; otherwise, it is invalid.

[0081] By acquiring on-site images of the target equipment and extracting actual operational data using visual processing algorithms, the digital twin model can obtain accurate data to re-predict faults. The validity of the first fault event is determined by comparing the results of the two fault predictions, thereby reducing the probability of incorrect fault predictions due to incomplete data. This improves the realism and accuracy of fault events triggered by the digital twin model, reduces the waste of human and material resources caused by erroneous fault predictions, and ultimately lowers maintenance costs. At this point, a fault investigation task for the first fault event is generated to facilitate on-site investigation by maintenance personnel.

[0082] Furthermore, in step 110, determining whether a data completion event was triggered in the digital twin model within the historical time period prior to the triggering of the first fault event may further include: sending a query request to the interaction module to receive the query result obtained by the interaction module querying the execution log of the data completion event in response to the query request; and determining whether a data completion event was triggered in the digital twin model within the historical time period prior to the triggering of the first fault event based on the query result.

[0083] Although the subscription module can receive various events triggered within the digital twin model, there is a priority between event subscription and message broadcasting during the actual configuration process. Therefore, even if the digital twin model triggers an event, it may not be received by the intelligent operation and maintenance system immediately. Thus, by having the intelligent operation and maintenance system actively query the digital twin model, the triggered events within the digital twin model can be obtained as soon as possible.

[0084] Furthermore, referring to Figure 3 In step 140, data extraction processing is performed on the on-site images to obtain the operating data of the target equipment, specifically including steps 141-143, wherein:

[0085] Step 141: Perform target recognition on the on-site images to obtain the image regions corresponding to each type of equipment;

[0086] Step 142: Determine the target image region from each image region. The target image region is the image region corresponding to the object device or an associated device that has a signal transmission relationship with the object device.

[0087] Step 143: Perform text recognition on the target image region to obtain the operating data of the target device.

[0088] Specifically, recognition results are obtained by identifying the on-site images. The recognition results include candidate boxes corresponding to each device, and the area corresponding to the candidate box in the on-site image is the image area corresponding to the device. Further, the type of the target device is determined. If the target device has a display screen, the image area corresponding to the target device is directly selected as the target image area. If the target device does not have a display screen, the image area corresponding to the associated device (edge ​​device) is used as the target image area.

[0089] Furthermore, text recognition is performed on the target image region to obtain a data form containing operational data corresponding to at least one device; then, based on the device type and data type of the target device, the operational data corresponding to the target device is determined from the data form.

[0090] Further, in step 140, sending the running data to the digital twin model may specifically include: converting the running data into a format to obtain standard data that is adapted to the input format of the fault prediction model; aligning the standard data with the data stream sent from the data mapping model to the fault prediction model in time; and sending the aligned standard data to the digital twin model.

[0091] When sending operational data to the digital twin model, the acquisition time of the on-site images must also be sent. The acquisition time of the on-site images is the time corresponding to the operational data of the target device, so that the extracted operational data of the target device can be aligned with the data of each device collected by the digital twin model along the time, thereby making the data available.

[0092] Furthermore, the preset time periods set in steps 150 and 160 are to limit the timeliness of the operational data fed back to the digital twin model and to distinguish the correlation between the second fault event retried by the digital twin model and the operational data. If the intelligent operation and maintenance system supplements the operational data of the target device to the digital twin model, but the digital twin model system triggers the second fault event only after a long interval, even if the second fault event corresponds to the same faulty device as the first fault event, it cannot be determined that the first fault event is valid. This is because due to the long interval, it is impossible to determine whether there is a causal relationship between the triggering of the second fault event and the supplemented operational data of the target device.

[0093] Therefore, the length of the preset time period should be set reasonably. In practice, it can be set by maintenance personnel based on experience. Of course, the length of the preset time period can also be determined through testing. First, historical fault data of the target system is collected, including historical operating data of each device in the target system during historical fault events. Each historical fault data is input into the data mapping model in the digital twin model to obtain the predicted fault output by the fault prediction model. Further, the digital twin model outputs the predicted time of the predicted fault for each historical fault data, and performs a weighted average of the predicted times to obtain the average time. The average time is used as the length of the preset time period.

[0094] Furthermore, since the prediction time is only used to test the speed at which the digital twin model processes fault predictions, rather than to determine its accuracy, there is no need to compare the predicted faults output by the digital twin model with the historical fault equipment corresponding to historical fault data, thus greatly improving the testing time.

[0095] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0096] Based on the same inventive concept, this application also provides a vision-based intelligent operation and maintenance device for implementing the vision-based intelligent operation and maintenance method described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more vision-based intelligent operation and maintenance device embodiments provided below can be found in the limitations of the vision-based intelligent operation and maintenance method described above, and will not be repeated here.

[0097] In one exemplary embodiment, such as Figure 4 As shown, a vision-based intelligent operation and maintenance device 400 is provided. The device includes an event determination module 401, an instruction generation module 402, an image acquisition module 403, a data extraction module 404, and a validity judgment module 405, wherein:

[0098] The event determination module 401 is used to determine whether a data completion event was triggered in the digital twin model during the historical time period before the first fault event was triggered, in the case where the digital twin model is determined to have triggered a first fault event.

[0099] The instruction generation module 402 is used to determine the target device corresponding to the data completion if it is determined that a data completion event has been triggered, and to generate an image acquisition instruction for the target device.

[0100] The image acquisition module 403 is used to send an image acquisition command to the target device to obtain the on-site image acquired by the target device in response to the image acquisition command.

[0101] The data extraction module 404 is used to extract and process data from the on-site images to obtain the operating data of the target equipment, and send the operating data to the digital twin model;

[0102] The validity judgment module 405 is used to determine that the first fault event is invalid if it is determined that the digital twin model has not triggered the second fault event within a preset time period, or if the second fault event triggered by the digital twin model corresponds to a different fault device than the first fault event.

[0103] If, within a preset time period, the digital twin model is found to trigger a second fault event that corresponds to the same fault device as the first fault event, then the first fault event is deemed valid, and a troubleshooting task for the first fault event is generated.

[0104] In the aforementioned intelligent operation and maintenance device based on visual algorithms, the validity (accuracy) of the first fault event triggered after the data completion event has been pre-triggered by the digital twin model is difficult to determine. Therefore, an image acquisition command is further generated for the target device corresponding to the data completion event. The target device acquires on-site images of the target device, and the visual algorithm extracts the operating data of the target device from the on-site images. The operating data is then fed back to the digital twin model, enabling the digital twin model to make a new fault prediction. If the digital twin model re-triggers a second fault event that is the same as the faulty device corresponding to the first fault event, then the first event is determined to be valid; otherwise, it is invalid.

[0105] By acquiring on-site images of the target equipment and extracting actual operational data using visual processing algorithms, the digital twin model can obtain accurate data to re-predict faults. The validity of the first fault event is determined by comparing the results of the two fault predictions, thereby reducing the probability of incorrect fault predictions due to incomplete data. This improves the realism and accuracy of fault events triggered by the digital twin model, reduces the waste of human and material resources caused by erroneous fault predictions, and ultimately lowers maintenance costs. At this point, a fault investigation task for the first fault event is generated to facilitate on-site investigation by maintenance personnel.

[0106] In one embodiment, the digital twin model includes an interaction module that completes missing data in the output data of the data mapping model and transmits the completed output data to the fault prediction model; the event determination module 401 is specifically used for:

[0107] Send a query request to the interaction module to receive the query results obtained by the interaction module in response to the query request by querying the execution log of the data completion event;

[0108] Based on the query results, determine whether a data completion event was triggered in the digital twin model during the historical time period prior to the triggering of the first fault event.

[0109] In one embodiment, the instruction generation module 402 is specifically used for:

[0110] Determine the equipment type and location of the target device;

[0111] Image acquisition instructions are generated based on device type and device location.

[0112] In one embodiment, the data extraction module 404 is specifically used for:

[0113] Target recognition is performed on the on-site images to obtain the image regions corresponding to each type of equipment;

[0114] The target image region is determined from each image region. The target image region is the image region corresponding to the object device or an associated device that has a signal transmission relationship with the object device.

[0115] Text recognition is performed on the target image region to obtain the operating data of the target device.

[0116] In one embodiment, the data extraction module 404 is specifically used for:

[0117] Perform text recognition on the target image region to obtain a data form containing operational data for at least one device;

[0118] Based on the device type and data type of the target device, determine the corresponding operating data of the target device from the data form.

[0119] In one embodiment, the data extraction module 404 is specifically used for:

[0120] Send operational data to the digital twin model, including:

[0121] The runtime data is converted to a standard format that is compatible with the input format of the fault prediction model.

[0122] The standard data and the data stream sent from the data mapping model to the fault prediction model are time-aligned, and the aligned standard data is then sent to the digital twin model.

[0123] In one embodiment, if it is determined that no data completion event was triggered in the digital twin model during the historical time period prior to the first fault event being triggered, then the first fault event is determined to be valid, and a fault investigation task for the first fault event is generated.

[0124] The modules in the aforementioned intelligent operation and maintenance device based on vision algorithms can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0125] In one exemplary embodiment, such as Figure 5 As shown, an intelligent operation and maintenance system is provided, which includes a target device, a subscription unit, a data interaction layer, a visual algorithm layer, and a perception layer, wherein:

[0126] The subscription unit is linked to the data interaction layer and the perception layer. The subscription unit is used to subscribe to various times that originate within the digital twin model.

[0127] The data interaction layer interacts with the target device and is linked to the visual algorithm layer. The data interaction layer is used to determine whether a data completion event was triggered in the digital twin model during the historical time period before the first fault event was triggered, in the case that the digital twin model has triggered a first fault event. If it is determined that a data completion event has been triggered, the target device corresponding to the completed data is identified, and an image acquisition command for the target device is generated. The image acquisition command is sent to the target device to obtain the on-site image acquired by the target device in response to the image acquisition command.

[0128] The target device is used to acquire on-site images of various devices within the system.

[0129] The visual algorithm layer is linked to the digital twin model and the perception layer. The visual algorithm layer is used to extract and process data from the on-site images to obtain the operating data of the object device, and then send the operating data to the digital twin model.

[0130] The perception layer is used to determine that the first fault event is invalid if it is determined that the digital twin model has not triggered the second fault event within a preset time period, or if the second fault event triggered by the digital twin model corresponds to a different fault device than the first fault event; if it is determined that the second fault event triggered by the digital twin model corresponds to the same fault device as the first fault event within a preset time period, the first fault event is determined to be valid, and a fault investigation task for the first fault event is generated.

[0131] The intelligent operation and maintenance system provided in this application embodiment can be applied to any of the methods described above in the intelligent operation and maintenance method embodiment based on vision algorithms.

[0132] In one exemplary embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, this electronic device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a vision-based intelligent operation and maintenance method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0133] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0134] In one exemplary embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described in the above embodiments of the intelligent operation and maintenance method based on vision algorithms.

[0135] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described in the above embodiments of the intelligent operation and maintenance method based on vision algorithms.

[0136] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0137] The steps of any one of the methods in the above embodiments of the intelligent operation and maintenance method based on vision algorithms.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An intelligent operation and maintenance method based on visual algorithms, characterized in that, The method is applied to an intelligent operation and maintenance system, and the method includes: If it is determined that the digital twin model has triggered a first fault event, determine whether a data completion event was triggered in the digital twin model during the historical time period prior to the first fault event being triggered; If the data completion event is determined to have been triggered, the target device corresponding to the completed data is determined, and an image acquisition command for the target device is generated. The image acquisition command is sent to the target device to obtain the on-site image acquired by the target device in response to the image acquisition command. The on-site images are processed to extract data, and the operating data of the target device is obtained. The operating data is then sent to the digital twin model. If it is determined that the digital twin model does not trigger the second fault event within a preset time period, or the second fault event triggered by the digital twin model corresponds to a different fault device than the first fault event, then the first fault event is determined to be invalid. If, within a preset time period, the digital twin model is determined to trigger the same faulty device as the first fault event, then the first fault event is determined to be valid, and a troubleshooting task for the first fault event is generated.

2. The method according to claim 1, characterized in that, The digital twin model includes an interaction module, which completes the missing data in the output data of the data mapping model and transmits the completed output data to the fault prediction model. Determining whether a data completion event was triggered within the digital twin model during the historical time period prior to the triggering of the first fault event includes: Send a query request to the interaction module to receive the query results obtained by the interaction module in response to the query request by querying the execution log of the data completion event; Based on the query results, determine whether a data completion event was triggered within the digital twin model during the historical time period prior to the first fault event being triggered.

3. The method according to claim 2, characterized in that, The generation of image acquisition instructions for the target device includes: Determine the device type and location of the target device; An image acquisition command is generated based on the device type and the device location.

4. The method according to claim 3, characterized in that, The step of extracting and processing data from the on-site images to obtain the operating data of the target device includes: Target recognition is performed on the on-site images to obtain the image regions corresponding to each type of device; A target image region is determined from each of the image regions, wherein the target image region is the image region corresponding to the object device or an associated device that has a signal transmission relationship with the object device; The target image region is subjected to text recognition to obtain the operating data of the target device.

5. The method according to claim 4, characterized in that, The step of performing text recognition on the target image region to obtain the operating data of the target device includes: Perform text recognition on the target image region to obtain a data form containing operational data corresponding to at least one device; Based on the device type and data type of the object device, the corresponding operating data of the object device is determined from the data form.

6. The method according to claim 4, characterized in that, Sending the operational data to the digital twin model includes: The operational data is converted to obtain standard data that is compatible with the input format of the fault prediction model. The standard data and the data stream sent from the data mapping model to the fault prediction model are time-aligned, and the aligned standard data is then sent to the digital twin model.

7. The method according to claim 1, characterized in that, The method further includes determining that the first fault event is valid if it is determined that the data completion event was not triggered in the digital twin model within the historical time period prior to the first fault event being triggered, and generating a fault investigation task for the first fault event.

8. An intelligent operation and maintenance device based on visual algorithms, characterized in that, The device includes an event determination module, an instruction generation module, an image acquisition module, a data extraction module, and a validity judgment module, wherein: The event determination module is used to determine whether a data completion event was triggered in the digital twin model during the historical time period before the first fault event was triggered, in the case where the digital twin model is determined to have triggered a first fault event. The instruction generation module is used to determine the target device corresponding to the data completion if the data completion event is determined to be triggered, and to generate an image acquisition instruction for the target device. An image acquisition module is used to send the image acquisition command to the target device to obtain the on-site image acquired by the target device in response to the image acquisition command. The data extraction module is used to extract and process data from the on-site images to obtain the operating data of the target device, and send the operating data to the digital twin model; The validity determination module is used to determine that the first fault event is invalid if it is determined that the digital twin model has not triggered the second fault event within a preset time period, or if the second fault event triggered by the digital twin model corresponds to a different fault device than the first fault event. If, within a preset time period, the digital twin model is determined to trigger the same faulty device as the first fault event, then the first fault event is determined to be valid, and a troubleshooting task for the first fault event is generated.

9. An intelligent operation and maintenance system, characterized in that, The system includes a target device, a subscription unit, a data interaction layer, a visual algorithm layer, and a perception layer, wherein: The subscription unit is linked to the data interaction layer and the perception layer. The subscription unit is used to subscribe to various events triggered within the digital twin model. The data interaction layer interacts with the target device and is linked to the visual algorithm layer. When a first fault event is triggered in the digital twin model, the data interaction layer determines whether a data completion event was triggered within the digital twin model during a historical time period prior to the first fault event. If the data completion event was triggered, the layer identifies the target device corresponding to the completed data and generates an image acquisition command for that target device. The image acquisition command is then sent to the target device to obtain the on-site image acquired by the target device in response to the image acquisition command. The target device is used to collect on-site images of various devices within the system; The visual algorithm layer is linked to the digital twin model, and the visual algorithm layer is linked to the perception layer. The visual algorithm layer is used to perform data extraction processing on the on-site image to obtain the operating data of the object device, and send the operating data to the digital twin model. The perception layer is used to determine that the first fault event is invalid if it is determined that the digital twin model has not triggered the second fault event within a preset time period, or if the second fault event triggered by the digital twin model corresponds to a different fault device than the first fault event; and to determine that the first fault event is valid if it is determined that the second fault event triggered by the digital twin model corresponds to the same fault device as the first fault event within a preset time period, and to generate a fault investigation task for the first fault event.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images