Fault prediction method and device for inspection system based on image data fusion

Abstract

The present application provides a fault prediction method and device for an inspection system based on image data fusion. The fault prediction method includes: obtaining multidimensional data of a first device node, the multidimensional data being data from multiple data sources within a historical time; determining the amount of change and change trend corresponding to each dimensional data in the multidimensional data based on the multidimensional data; inputting the multidimensional data and the amount of change and change trend corresponding to each dimensional data into a first neural network model for fusion processing to obtain a prediction result, which is used to characterize the probability of occurrence of a first fault; in response to the probability of occurrence of the first fault being greater than a preset probability threshold, issuing an early warning notification of the first fault. The embodiments of the present application help to improve the accuracy of fault prediction and reduce the labor intensity of staff.

Description

Technical Field

[0001] The present application relates to the field of equipment detection, and in particular to a fault prediction method and device for an inspection system based on image data fusion. Background Art

[0002] Tunnels are mainly composed of water supply pipelines, cables, drainage, ventilation, power supply, lighting, fire protection, communications and other power transmission equipment and ancillary equipment. Tunnels include a large number of equipment and many distribution points, which brings heavy inspection content to manual inspections. The safe production of the water supply system in the tunnel is an important link in the overall safe and economic production of the water supply pump station. If there are safety hazards in the water supply system, such as changes in the tunnel structure, cracks in the tunnel, water seepage, etc., it will cause the water supply efficiency to be reduced due to accidents such as equipment damage, and it will not be able to continuously supply water to users, resulting in significant economic losses.

[0003] At present, most water supply tunnel inspections are mainly completed by manual inspections, and the various types of defects and abnormalities in water supply tunnels are basically judged manually. However, manual inspections are time-consuming and labor-intensive, the detection precision is insufficient, and it is impossible to make intelligent judgments on the real-time status of the equipment, and it is impossible to achieve key monitoring of key equipment or parts. There is a problem that maintenance after a failure will have serious consequences. Summary of the invention

[0004] The present application provides a fault prediction method and device for an inspection system based on image data fusion. The various aspects involved in the present application are introduced below.

[0005] In the first aspect, the present application provides a fault prediction method for an inspection system based on image data fusion, wherein the inspection system is an inspection system for a tunnel water supply pipeline, and the method comprises: obtaining multidimensional data of a first device node, wherein the multidimensional data is data from multiple data sources within a historical time, and the multiple data sources include at least the following multiple sensors: a visible light camera, an ultrasonic flaw detector, a temperature sensor, and a humidity sensor; determining, based on the multidimensional data, a change amount and a change trend corresponding to each dimensional data in the multidimensional data; inputting the multidimensional data and the change amount and change trend corresponding to each dimensional data into a first neural network model for fusion processing to obtain a prediction result, and the prediction result is used to characterize the probability of occurrence of a first fault; in response to the probability of occurrence of the first fault being greater than a preset probability threshold, issuing an early warning notification of the first fault; wherein the first fault is any one of the following multiple faults: rust, thread displacement, tunnel joint leakage, tunnel structure change, and tunnel crack.

[0006] In a second aspect, the present application provides a fault prediction device for an inspection system based on image data fusion, wherein the inspection system is an inspection system for a tunnel water supply pipeline, and the device includes: an acquisition module, used to acquire multidimensional data for a first device node, wherein the multidimensional data is data from multiple data sources within a historical time, and the multiple data sources include at least the following multiple sensors: a visible light camera, an ultrasonic flaw detector, a temperature sensor, and a humidity sensor; a determination module, used to determine, based on the multidimensional data, the change amount and change trend corresponding to each dimensional data in the multidimensional data; a prediction module, used to input the multidimensional data, the change amount and change trend corresponding to each dimensional data into a first neural network model for fusion processing to obtain a prediction result, and the prediction result is used to characterize the probability of occurrence of a first fault; a notification module, used to issue an early warning notification of the first fault in response to the probability of occurrence of the first fault being greater than a preset probability threshold; wherein the first fault is any one of the following multiple faults: rust, thread displacement, tunnel joint leakage, tunnel structure change, and tunnel crack.

[0007] In a third aspect, the present application provides an inspection system based on image data fusion, comprising: a memory for storing code; a processor connected to the memory, for executing the code stored in the memory, so that the inspection system performs the method described in the first aspect.

[0008] In a fourth aspect, the present application provides a non-volatile computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it can implement the method described in the first aspect.

[0009] In a fifth aspect, the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to implement the method described in the first aspect.

[0010] In the embodiment of the present application, based on the multidimensional data of the first device node in the historical time, the change amount and change trend of each dimensional data are determined, and according to the multidimensional data and its change amount and change trend, the data is fused and processed through the neural network model to determine the probability of the first fault, and when the probability of the first fault is greater than the preset probability threshold, an early warning notification of the first fault is issued. In the embodiment of the present application, by automatically collecting multidimensional data, statistically analyzing inspection data, and predicting the probability of the first fault according to the multidimensional data and its change amount and change trend, the accuracy of fault prediction is improved. It helps to reduce the labor intensity of staff, reduce the number of special inspections, and save the labor cost of enterprises. In the embodiment of the present application, fault prediction and monitoring are realized by automated analysis, and the introduction of dynamic early warning helps to find problems in time and carry out preventive maintenance in advance, and a more effective maintenance plan can be formulated and implemented in advance. Thereby avoiding serious damage caused by potential faults and avoiding higher maintenance costs caused by hasty handling of fault problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0012] Figure 1 It is a flow chart of a fault prediction method for an inspection system based on image data fusion provided in an embodiment of the present application.

[0013] Figure 2 yes Figure 1 A flowchart of a possible implementation of the method.

[0014] Figure 3 It is a schematic diagram of the composition of a fault prediction device of an inspection system based on image data fusion provided in an embodiment of the present application.

[0015] Figure 4 It is a schematic diagram of the composition of an inspection system based on image data fusion provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The same or similar reference numerals are used in the drawings to represent the same or similar modules. It should be understood that the drawings are only schematic, and the protection scope of the present application is not limited thereto.

[0017] Tunnels are mainly composed of water supply pipelines, cables, drainage, ventilation, power supply, lighting, fire protection, communications and other power transmission equipment and ancillary equipment. Tunnels include a large number of equipment and many distribution points, which brings heavy inspection content to manual inspections. At the same time, tunnels are prone to water accumulation and accumulation of harmful gases, and the internal air circulation is poor, which puts higher requirements on daily inspections and increases the danger of manual inspections. For example, the production system of the Zhanjiang Bay Shield Tunnel is mainly composed of water supply devices and supporting equipment, among which the safe production of the water supply system in the tunnel is an important part of the overall safe and economic production of the water supply pump station.

[0018] If there are safety hazards in the water supply system, such as changes in tunnel structure, cracks in the tunnel, water seepage, etc., the water supply efficiency will be reduced due to equipment damage and other accidents, and water cannot be continuously supplied to users, resulting in significant economic losses. Therefore, the safe and stable operation of the main equipment and auxiliary equipment of each system in the water supply tunnel directly affects the safety and production level of the water supply station.

[0019] At present, power tunnels with similar scenes at home and abroad have begun to use fixed-point monitoring equipment to detect tunnel environmental conditions. However, fixed monitoring has poor flexibility and has blind spots, which cannot meet the requirements of tunnel inspection. At present, most water supply tunnel inspections are mainly completed by manual inspections, and the various types of defects and abnormalities in water supply tunnels are basically judged manually.

[0020] Due to the large scale of tunnel inspection, the large number of inspection objects and the poor inspection environment, manual inspection has the following main problems: (1) Manual inspection mainly relies on sensory judgment, which is subjective and limited by personal experience, so the inspection quality varies; (2) Manual inspection is highly dependent on personnel and cannot be available at any time; (3) The inspection results of manual inspection are restricted by the inspection methods, data recording and data analysis. Usually, a lot of manpower is spent but effective data results cannot be obtained, and the obtained data is difficult to form a system and has little reference value; (4) Manual inspection cannot view equipment data in real time, cannot find and eliminate faults in time, and the inspection efficiency is low; (5) Manual inspection is time-consuming and labor-intensive, with many inspection contents and high labor intensity; (6) The tunnel operating environment is poor, and there are certain health risks for workers to conduct inspections in these areas for a long time, and once an accident occurs, it is easy to cause personnel safety accidents. (7) It is difficult to achieve key monitoring of key equipment or parts, and it is difficult to achieve comprehensive protection.

[0021] It can be seen that in the current inspection of water supply tunnels, there is a problem that the manual detection is not precise enough, the real-time status of the equipment cannot be intelligently judged, key equipment or parts cannot be monitored, and maintenance after a failure has serious consequences.

[0022] Therefore, in order to solve one or more of the above-mentioned problems, it is necessary to design a technical solution for a water supply tunnel inspection system with good performance.

[0023] Based on this, the present application embodiment proposes a fault prediction method for an inspection system based on image data fusion. The fault prediction method for an inspection system based on image data fusion in the present application embodiment is applied to a water supply tunnel, that is, the inspection system is an inspection system for a water supply tunnel. Figure 1 The fault prediction method of the inspection system based on image data fusion according to the embodiment of the present application is described in detail. Figure 1 As shown, the fault prediction method of the inspection system based on image data fusion in the embodiment of the present application may mainly include steps S110 to S140, which are described in detail below.

[0024] It should be pointed out that the size of the serial numbers of the steps in the embodiments of the present application does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0025] In step S110, multi-dimensional data of the first device node is obtained. The multi-dimensional data is data from multiple data sources in historical time, and the multiple data sources include at least the following multiple sensors: visible light camera, ultrasonic flaw detector, temperature sensor, humidity sensor.

[0026] The water supply tunnel may include multiple device nodes, and important multiple device nodes may be, for example, tunnel joints and threaded connections. The first device node is any device node among the multiple device nodes, and the first device node is a node for which detection data needs to be collected.

[0027] In some implementations, the plurality of device nodes may also include: meters, switches, indicator lights, etc. of electrical equipment, so as to perform meter readings, switch status monitoring, indicator light status monitoring, etc. of the electrical equipment.

[0028] Water supply tunnels have their own particularities. The internal operating environment is relatively complex, usually dark and humid, and the air circulation is poor. In some implementations, in order to obtain real-time environmental information inside the tunnel and adapt to various fault detections, multiple sensors may include but are not limited to temperature sensors, humidity sensors, visible light cameras, ultrasonic flaw detectors, harmful gas detection sensors, smoke detection sensors, and light detection sensors.

[0029] The multiple data sources include at least the following multiple sensors: visible light camera, ultrasonic flaw detector, temperature sensor, humidity sensor. For example, the multi-dimensional data may include image dimension data, ultrasonic flaw detector dimension data, temperature dimension data, and humidity dimension data.

[0030] In some implementations, the multiple sensors may also include: a thermal imager, an ultraviolet detector, a fill light device, etc.

[0031] In some implementations, the multi-dimensional data of the first device node can be acquired by a patrol robot. It is understandable that the patrol robot is equipped with multiple sensors for acquiring multi-dimensional data, and can automatically acquire the required detection data of each dimension at the first device node. It should be noted that the data collection of the first device node may be continuous and periodic. For example, after the patrol robot collects the data, it stores the collected data in a fixed path or in a remote server, and the execution body of the patrol system can be the server. In other embodiments, the execution body of the patrol system can also be a patrol robot.

[0032] In step S120, based on the multidimensional data, the change amount and change trend corresponding to each dimension of the multidimensional data are determined.

[0033] For example, based on the data of the visible light camera dimension, the change amount and the change trend over time corresponding to the data of the visible light camera dimension are determined. Based on the data of the temperature sensor dimension, the change amount and the change trend over time corresponding to the data of the temperature sensor dimension are determined.

[0034] By comparing the characteristics of historical data, we can to a certain extent reflect the future development and changes of the equipment, that is, whether a failure is likely to occur.

[0035] In step S130, the multidimensional data, the change amount and the change trend corresponding to each dimensional data are input into the first neural network model for fusion processing to obtain a prediction result. The prediction result is used to characterize the probability of occurrence of the first fault.

[0036] Among them, the first fault is one of the following faults: rust, thread displacement, tunnel joint leakage, tunnel structure change, and tunnel crack.

[0037] The inspection system may include multiple neural network models, and the first neural network model is any one of the multiple neural network models. The first neural network model is used to predict the first fault, or in other words, the first neural network model corresponds to the first fault.

[0038] For pattern recognition tasks, an efficient deep learning model is required. The first neural network model can be a convolutional neural network (CNN), a recurrent neural network (RNN), R-CNN, Transformer, Resnet, and Supernet.

[0039] Using the pre-prepared multidimensional data set, the first neural network model is allowed to learn how to identify and predict the characteristics of the first fault through multiple rounds of iterations, while optimizing the network parameters to achieve the best identification and prediction effects. In this process, the parameters of the first neural network model are adjusted, such as the four important dimensions of the convolutional neural network architecture, namely depth, width, kernel size, and resolution.

[0040] The characteristics of the first fault may include multiple parameters, such as basic parameters of tunnel cracks, which may include one or more of crack spacing, opening, extension length, strike, inclination and dip.

[0041] Since multiple types of sensor data are used, including multiple different types of faults, it is necessary to fuse the information of different sensor data corresponding to multiple fault models. Specifically, sensor-level fusion methods, such as sensor fusion networks, can be used to integrate the observation results of different sensors into a consistent representation. Due to the different dimensions and data types of each dimensional data, its data format may not meet the input requirements of some neural network models. In some implementations, before the multidimensional data, the corresponding changes and time-varying trends of each dimensional data are input into the first neural network model, the multidimensional data, the corresponding changes and time-varying trends of each dimensional data, etc. can be processed, such as batch normalization, to meet the input requirements of the first neural network model. In addition, the batch normalization operation can make the distribution of features of each layer of the network as stable as possible, suppress gradient explosion, and speed up training.

[0042] In some implementations, the input of the first neural network model may also include a first parameter, which is a weight corresponding to each dimension of the multidimensional data. In step S130, the multidimensional data, the change amount and change trend corresponding to each dimension of the data are input into the first neural network model for fusion processing to obtain a prediction result, which may include: inputting the multidimensional data, the change amount and change trend corresponding to each dimension of the data, and the first parameter into the first neural network model for fusion processing to obtain a prediction result. The contribution and importance of each dimension of data to the prediction result are different, and different weights are assigned, which is conducive to improving the precision and accuracy of the fault prediction result.

[0043] In some implementations, the input of the first neural network model may also include a second parameter, which is the ratio of the used time of the first device node to the rated service life. The above-mentioned inputting the multidimensional data, the change amount and change trend corresponding to each dimensional data, and the first parameter into the first neural network model for fusion processing to obtain the prediction result may include: inputting the multidimensional data, the change amount and change trend corresponding to each dimensional data, the first parameter, and the second parameter into the first neural network model for fusion processing to obtain the prediction result. In this way, combining the use time is conducive to further improving the precision and accuracy of the prediction results.

[0044] In step S140, in response to the probability of occurrence of the first fault being greater than a preset probability threshold, a warning notification of the first fault is issued.

[0045] Specifically, the probability of the occurrence of the fault is calculated through multi-dimensional data, and then a fault warning is issued based on whether the calculated probability value exceeds the alarm threshold. For example, the probability that the first device node A generates the first fault (such as tunnel joint leakage) is P = F (a, b, c, d), a, b, c, d are the four elements that cause tunnel joint leakage (image data, ultrasonic flaw detector data, temperature data, humidity data), and F () is the Transformer network. When P> the preset probability, the first device node is notified of the first fault. For example, the text or image of the first fault can be displayed on the display device of the system, and a subsequent safety measure plan can be given. The inspection personnel can work according to the plan to replace or eliminate the danger.

[0046] In some implementations, the first fault can be divided into severe, moderate, and mild levels according to the severity. The preset probability threshold includes a preset first probability threshold, a second probability threshold, and a third probability threshold. The first probability threshold, the second probability threshold, and the third probability threshold correspond to severe first faults, moderate first faults, and mild first faults, respectively. In response to the probability of occurrence of the first fault being greater than the preset probability threshold, issuing a warning notification of the first fault in step S140 may include: issuing a warning notification of severe first fault in response to the probability of occurrence of the first fault being greater than the preset first probability threshold; issuing a warning notification of moderate first fault in response to the probability of occurrence of the first fault being greater than the preset second probability threshold; issuing a warning notification of mild first fault in response to the probability of occurrence of the first fault being greater than the preset third probability threshold. By grading the first fault, it is helpful to improve the refined management of fault prediction in inspection.

[0047] In the embodiment of the present application, based on the multidimensional data of the first device node in the historical time, the change amount and change trend of each dimensional data are determined, and according to the multidimensional data and its change amount and change trend, the data is fused and processed through the neural network model to determine the probability of the first fault, and when the probability of the first fault is greater than the preset probability threshold, an early warning notification of the first fault is issued. In the embodiment of the present application, by automatically collecting multidimensional data, statistically analyzing inspection data, and predicting the probability of the first fault according to the multidimensional data and its change amount and change trend, the accuracy of fault prediction is improved. It helps to reduce the labor intensity of staff, reduce the number of special inspections, and save the labor cost of enterprises. In the embodiment of the present application, fault prediction and monitoring are realized by automated analysis, and the introduction of dynamic early warning helps to find problems in time and carry out preventive maintenance in advance, and a more effective maintenance plan can be formulated and implemented in advance. Thereby avoiding serious damage caused by potential faults and avoiding higher maintenance costs caused by hasty handling of fault problems.

[0048] The method of the embodiment of the present application is further explained below in combination with some possible implementation methods of the present application.

[0049] Under different lighting conditions, the acquired image data may be different, thus affecting the accuracy of the prediction results. In some implementations, the image data in the multidimensional data is the data captured by a visible light camera under a preset brightness. The same brightness (illuminance) range helps to ensure the stability of the image data. During the image data acquisition process, fill-in lighting devices, such as lighting lamps, reflectors, etc., can be used to adjust the illumination so that the brightness during acquisition meets the preset brightness conditions, thereby improving the accuracy of the image data and the recognition accuracy. The fill-in lighting device can be installed at the inspection robot or at each device node.

[0050] Since there are many different types of faults, the inspection system may include multiple neural network models, and the first neural network model is any one of the multiple neural network models. A hyper-network usually includes subnets of all possible structures. When testing the performance of a certain network structure, the parameters trained by the hyper-network can be directly inherited. The first neural network model corresponding to the first fault can be obtained through hyper-network training.

[0051] In some implementations, the supernet can be built based on MobileNetV2. The depthwise separable module of MobileNetV2 includes two independent parts: depthwise convolution and pointwise convolution, both of which add batch normalization layers and nonlinear activation units to the output results. Depthwise separable convolution is different from conventional convolution operations. One convolution kernel of depthwise separable convolution is responsible for one channel, and one channel is convolved by only one convolution kernel.

[0052] In some implementations, the supernet can be trained by progressive shrinkage. The network training can be performed in a mixed loss manner, and the mixed loss can include regression loss and classification loss. The process of progressive shrinkage training of the supernet model can mainly include: determining the regression loss and classification loss of the supernet model; updating the parameters of the supernet model according to the regression loss and classification loss. Optionally, the regression loss can use SmoothL1 loss, L1 loss, L2 loss, Huber loss, etc. Optionally, the classification loss can use binary cross entropy (BCE) and Focal loss. For example, in BCE, when the predicted value is close to the label value, the loss is small, and when the predicted value is far from the label value, the loss is large. This feature is conducive to model learning.

[0053] Figure 2 yes Figure 1 A flowchart of a possible implementation of the method is shown in FIG. 1 : A first neural network model corresponding to a first fault is obtained through supernet training. Figure 2 The method may include steps S210 to S230, which are described in detail below.

[0054] In step S210, the pre-trained MobileNetV2 is used as the supernet and trained as the baseline model.

[0055] In step S220, the supernet is trained using a progressive shrinkage method.

[0056] Progressive shrinking can be divided into multiple stages, and the supernet is trained by progressive shrinking according to different dimensions. The precision r, kernel size k, depth d, and width w can be adjusted separately.

[0057] During training, each generation of neural network training (epoch) is a sub-network randomly sampled from the sampling space, and the sub-network shares weights with the super-network. The epoch uses all the data in the training set to fully train the model. For example, the sampling space can be: Φ = {r∈{168,224,256,512}, k∈{3,5,7}, d∈{1,2,3}, w∈{0.8,0.6,1.0}}. Among them, r represents the resolution, k represents the convolution kernel size, d represents the number of convolution blocks in a convolution unit, and w represents the ratio of the number of convolution kernel output channels to the maximum number of channels.

[0058] In step S230, the subnet is searched for evolutionarily.

[0059] An evolutionary search algorithm may be used to search for a subnet that meets a preset accuracy requirement as the first neural network model.

[0060] Combination of the above Figure 1-Figure 2 The method embodiment of the present application is described in detail. Figure 3 to Figure 4 The device embodiment of the present application is described in detail. It should be understood that the description of the device embodiment corresponds to the description of the method embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0061] The embodiment of the present application also provides a fault prediction device for an inspection system based on image data fusion. Figure 3 1 is a schematic diagram of the composition of a fault prediction device for an inspection system based on image data fusion provided in an embodiment of the present application, wherein the inspection system is an inspection system for a tunnel water supply pipeline. Figure 3 As shown, the fault prediction device 300 of the inspection system based on image data fusion may include: an acquisition module 310 , a determination module 320 , a prediction module 330 , and a notification module 340 .

[0062] The acquisition module 310 is used to acquire multi-dimensional data for the first device node. The multi-dimensional data is data from multiple data sources in historical time, and the multiple data sources include at least the following multiple sensors: visible light camera, ultrasonic flaw detector, temperature sensor, humidity sensor.

[0063] The determination module 320 is used to determine the change amount and change trend corresponding to each dimension of the multidimensional data according to the multidimensional data.

[0064] The prediction module 330 is used to input the multi-dimensional data, the change amount and the change trend data corresponding to each dimensional data into the first neural network model for fusion processing to obtain a prediction result. The prediction result is used to characterize the probability of occurrence of the first fault.

[0065] The notification module 340 is used to issue a warning notification of the first fault in response to the probability of occurrence of the first fault being greater than a preset probability threshold, wherein the first fault is any one of the following faults: corrosion, thread displacement, tunnel joint leakage, tunnel structure change, and tunnel crack.

[0066] Optionally, the input of the first neural network model also includes a first parameter, which is a weight corresponding to each dimension of the multidimensional data. The prediction module 330 is used to input the multidimensional data, the change amount and change trend corresponding to each dimension of the data, and the first parameter into the first neural network model for fusion processing to obtain a prediction result.

[0067] Optionally, the input of the first neural network model also includes a second parameter, which is the ratio of the used time of the first device node to the rated service life. The prediction module 330 is used to input the multidimensional data, the change amount and change trend corresponding to each dimensional data, the first parameter, and the second parameter into the first neural network model for fusion processing to obtain a prediction result.

[0068] Optionally, the first fault is divided into severe, moderate, and mild levels according to the severity, and the preset probability thresholds include a preset first probability threshold, a second probability threshold, and a third probability threshold. The notification module 340 is used to issue a warning notification of a severe first fault in response to the probability of the first fault being greater than the preset first probability threshold; to issue a warning notification of a moderate first fault in response to the probability of the first fault being greater than the preset second probability threshold; and to issue a warning notification of a mild first fault in response to the probability of the first fault being greater than the preset third probability threshold.

[0069] Optionally, the image data in the multi-dimensional data is data captured by a visible light camera under preset brightness.

[0070] The embodiment of the present application automatically collects multi-dimensional data, statistically analyzes inspection data, and predicts the probability of the first fault according to the multi-dimensional data and its change amount and change trend, which helps to improve the accuracy of fault prediction. It helps to reduce the labor intensity of staff, reduce the number of special inspections, and save the company's labor costs. The embodiment of the present application uses automated analysis to achieve fault prediction and monitoring, and the introduction of dynamic early warning helps to detect problems in a timely manner and perform preventive maintenance in advance, and a more effective maintenance plan can be formulated and implemented in advance. This avoids serious damage caused by potential faults and avoids higher maintenance costs caused by hasty handling of fault problems.

[0071] Figure 4 Schematic diagram of component units / partial component units of the inspection system based on image data fusion provided in the embodiment of the present application. Figure 4As shown, the inspection system 400 based on image data fusion may include: a memory 410 and at least one processor 420 .

[0072] The memory 410 is used to store codes or computer programs.

[0073] The processor 420 is connected to the memory 410 and is used to execute the code or computer program stored in the memory 410 to control the inspection system 400 based on image data fusion to execute any of the methods described above.

[0074] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 410 and executed by the processor 420 to complete the present application.

[0075] Those skilled in the art will understand that Figure 4 The above is only an example of the inspection system 400 based on image data fusion, and does not constitute a limitation on the inspection system based on image data fusion. The system may include more or fewer components than shown in the figure, or some components may be combined, or different components may be included.

[0076] The processor 420 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0077] The inspection system 400 based on image data fusion provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0078] An embodiment of the present application further provides a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0079] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0080] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0081] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above method embodiments, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above method embodiments can be implemented. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), read-only compact disc (CD-ROM), magnetic tape, floppy disk and optical data storage device. The computer-readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, it can be a non-transient storage medium.

[0082] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0083] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0085] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0086] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0087] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A fault prediction method for an inspection system based on image data fusion, characterized in that: The inspection system is an inspection system for a water supply tunnel, and the fault prediction method includes: Acquire multidimensional data of the first device node, where the multidimensional data is data from multiple data sources in a historical time, and the multiple data sources include at least the following multiple sensors: a visible light camera, an ultrasonic flaw detector, a temperature sensor, and a humidity sensor; Determine, based on the multidimensional data, a change amount and a change trend corresponding to each dimensional data in the multidimensional data; Inputting the multidimensional data, the change amount and the change trend corresponding to each dimensional data into the first neural network model for fusion processing to obtain a prediction result, wherein the prediction result is used to characterize the probability of occurrence of the first fault; In response to the probability of occurrence of the first fault being greater than a preset probability threshold, issuing a warning notification of the first fault; Among them, the first fault is one of the following faults: rust, thread displacement, tunnel joint leakage, tunnel structure change, and tunnel crack.

2. The fault prediction method according to claim 1, characterized in that: The input of the first neural network model also includes a first parameter, which is a weight corresponding to each dimensional data. The multidimensional data and the change amount and change trend corresponding to each dimensional data are input into the first neural network model for fusion processing to obtain a prediction result, including: The multidimensional data, the change amount and change trend corresponding to each dimensional data, and the first parameter are input into the first neural network model for fusion processing to obtain the prediction result.

3. The fault prediction method according to claim 2, characterized in that: The input of the first neural network model also includes a second parameter, which is the ratio of the used time of the first device node to the rated service life. The multidimensional data, the change amount and change trend corresponding to each dimensional data, and the first parameter are input into the first neural network model for fusion processing to obtain the prediction result, including: The multidimensional data, the change amount and change trend corresponding to each dimensional data, the first parameter, and the second parameter are input into the first neural network model for fusion processing to obtain the prediction result.

4. The fault prediction method according to claim 1, characterized in that: The first fault is divided into severe, moderate and mild levels according to the severity, the preset probability threshold includes a preset first probability threshold, a second probability threshold and a third probability threshold, and in response to the probability of occurrence of the first fault being greater than the preset probability threshold, issuing an early warning notification of the first fault includes: In response to the probability of occurrence of the first fault being greater than a preset first probability threshold, issuing a warning notification of a severe first fault; In response to the probability of occurrence of the first fault being greater than a preset second probability threshold, issuing a moderate early warning notification of the first fault; In response to the probability of the first fault occurring being greater than a preset third probability threshold, an early warning notification of a minor first fault is issued.

5. The fault prediction method according to claim 1, characterized in that: The image data in the multi-dimensional data is data captured by the visible light camera under a preset brightness.

6. A fault prediction device for an inspection system based on image data fusion, characterized in that: The inspection system is an inspection system for a tunnel water supply pipeline, and the fault prediction device comprises: An acquisition module, configured to acquire multidimensional data for a first device node, wherein the multidimensional data is data from multiple data sources within a historical period, wherein the multiple data sources include at least the following multiple sensors: a visible light camera, an ultrasonic flaw detector, a temperature sensor, and a humidity sensor; A determination module, used to determine, based on the multidimensional data, a change amount and a change trend corresponding to each dimensional data in the multidimensional data; A prediction module, used for inputting the multi-dimensional data, the change amount and the change trend data corresponding to each dimensional data into a first neural network model for fusion processing to obtain a prediction result, wherein the prediction result is used to characterize the probability of occurrence of the first fault; a notification module, configured to issue a warning notification of the first fault in response to the probability of occurrence of the first fault being greater than a preset probability threshold; The first fault is any one of the following faults: rust, thread displacement, tunnel joint leakage, tunnel structure change, and tunnel crack.

7. The device according to claim 6, characterized in that The input of the first neural network model also includes a first parameter, which is the weight corresponding to the data of each dimension. The prediction module is used to input the multidimensional data, the change amount and change trend corresponding to the data of each dimension, and the first parameter into the first neural network model for fusion processing to obtain the prediction result.

8. The device according to claim 7, characterized in that The input of the first neural network model also includes a second parameter, which is the ratio of the used time of the first device node to its rated service life. The prediction module is used to input the multidimensional data, the change amount and change trend corresponding to the data of each dimension, the first parameter, and the second parameter into the first neural network model for fusion processing to obtain the prediction result.

9. An inspection system based on image data fusion, characterized in that: include: A memory for storing codes; A processor is connected to the memory and is used to execute the code stored in the memory so that the inspection system executes the fault prediction method according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the computer program is used to implement the fault prediction method according to any one of claims 1 to 5.

Images