Multi-physics field fusion anomaly identification method and system, electronic device, and storage medium

Abstract

The application discloses a kind of multi-physical field fusion's abnormal identification method, comprising the following steps: obtaining monitoring object sample video, abnormal condition sample video and real-time monitoring video, abnormal condition sample video is multi-physical field video;Target equipment is labeled to monitoring object sample video and is extracted separately in turn, and first convolutional neural network model is trained, and target equipment identification model is established;Abnormal condition sample video is labeled and abnormal coordinate point is extracted, and second convolutional neural network model is trained, and abnormal condition identification model is established;Abnormal coordinate point is extracted to real-time monitoring video, input the model established, and obtain abnormal identification result.The application further discloses a kind of multi-physical field fusion's abnormal identification system, electronic equipment and storage medium.The application carries out fusion processing to multi-physical field video, realizes the quick identification and positioning of abnormal state in real-time monitoring video by double convolutional neural network model.

Description

Technical Field

[0001] This invention relates to the field of equipment abnormality monitoring technology, and in particular to an anomaly identification method, system, electronic device and storage medium based on multi-physics field fusion. Background Technology

[0002] With increasingly stringent safety requirements, traditional leak detection technologies, such as fixed gas detectors, can no longer meet the growing safety demands of enterprises. Various new leak detection technologies are constantly being updated and iterated, among which imaging-based surface monitoring technologies, represented by optical and acoustic principles, are gaining increasing attention. Imaging-based surface monitoring technologies can directly observe abnormal conditions such as gas leaks, abnormal noise, partial discharge, and high-temperature flames through video. They offer advantages such as wide coverage, fast response speed, and visualized results, and have wide applications in leak detection, partial discharge detection, and flame detection. However, currently, the analysis of monitoring results from imaging-based surface monitoring technologies is still primarily manual, especially in scenarios requiring the integrated application of multiple technologies. When a single physical quantity cannot yield accurate results, manual analysis combining multiple physical quantity results is necessary. This results in low automation, low detection efficiency, and limitations imposed by the analyzers' time and energy, hindering large-scale deployment.

[0003] Patent document CN110266268A discloses a photovoltaic module fault detection method based on image fusion recognition. This method acquires infrared thermal imaging and visible light images of the photovoltaic module, and uses image processing and convolutional neural networks for classification and recognition to determine whether the photovoltaic module has malfunctioned. However, this method monitors only a single object type, judging anomalies solely from the perspective of temperature change, and uses only a single convolutional neural network model, lacking the ability to identify target types.

[0004] Patent document CN108008259A discloses a detection method and device based on the integrated fusion of infrared, ultraviolet, and visible light images. It fuses an equipment image with an infrared fault detection image to obtain an infrared detection equipment image, fuses the equipment image with an ultraviolet spot image to obtain an ultraviolet spot equipment image, and fuses the infrared detection equipment image with the ultraviolet spot equipment image to obtain a fused image used for fault determination of the detected equipment. However, this method only achieves pairwise combination of imaging results and does not realize automatic identification and alarm of abnormal states.

[0005] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] One of the objectives of this invention is to provide a multi-physics field fusion anomaly identification method, system, electronic device, and storage medium, thereby improving the problems of low monitoring efficiency and limited applicable scenarios of single-type monitoring devices.

[0007] Another objective of this invention is to provide a method, system, electronic device, and storage medium for anomaly identification based on multi-physics field fusion, thereby overcoming the reliance on manual analysis and judgment in existing imaging-based surface monitoring technologies.

[0008] To achieve the above objectives, according to a first aspect of the present invention, the present invention provides a multi-physics field fusion anomaly identification method, comprising the following steps: acquiring a monitoring object sample video, an abnormal operating condition sample video, and a real-time monitoring video, wherein the monitoring object sample video is a visible light video, the abnormal operating condition sample video is a multi-physics field video, and the real-time monitoring video includes both visible light video and multi-physics field video; labeling the monitoring object sample video with target equipment and extracting it sequentially and individually, training a first convolutional neural network model, and establishing a target equipment identification model; labeling the abnormal operating condition sample video with abnormal operating conditions and extracting abnormal coordinate points, training a second convolutional neural network model, and establishing an abnormal operating condition identification model; extracting abnormal coordinate points from the real-time monitoring video, inputting the abnormal operating condition identification model and the target equipment identification model, and obtaining an anomaly identification result.

[0009] Furthermore, in the above technical solution, the multiphysics field video includes infrared temperature measurement video, infrared gas imaging video, ultraviolet light distribution video, and sound field cloud map video.

[0010] Furthermore, the anomaly identification method of multi-physics field fusion in the above technical solution also includes the step of standardizing the sample video of the monitored object, the sample video of the abnormal working condition, and the real-time monitoring video.

[0011] Furthermore, in the above technical solutions, standardization includes resolution standardization, distortion correction, and image cropping.

[0012] Furthermore, in the above technical solutions, the types of target equipment include pumps, valves, transformers, pipelines, flanges, steel structures, pressure vessels, storage tanks, and towers; the types of abnormal operating conditions include leakage, vibration, noise, flame, and partial discharge.

[0013] Furthermore, in the above technical solution, when training the first convolutional neural network model, the sample video of the monitored object is extracted sequentially and separately according to the coordinate range of the target device, and divided into three channels of video (R, G, B) as input, and the type of the target device is used as the encoded output.

[0014] Furthermore, in the above technical solution, the extraction of abnormal coordinate points includes:

[0015] Infrared thermometry video, sound field cloud map video, ultraviolet light distribution video, and infrared gas imaging video are respectively used as T, S, U, and G four-channel videos;

[0016] Perform grayscale transformation on the four video channels: T, S, U, and G;

[0017] Extracting outlier coordinates from the T, S, U, and G channels of the video:

[0018] If there is a temperature anomaly, extract the coordinates (x, y) of the center point of the temperature anomaly area in the T-channel video. T ,y T );

[0019] If there is noise anomaly, extract the coordinates (x, y) of the center point of the noise region in the S-channel video. S ,y S );

[0020] If there is an ultraviolet light anomaly, extract the coordinates (x, y) of the center point of the ultraviolet light source in the U-channel video. U ,y U );as well as

[0021] If there is a gas anomaly, extract the coordinates (x, y) of the center point of the gas cloud region in the G channel video. G ,y G );

[0022] By comparing the coordinate values, a coordinate matrix is ​​obtained. in,

[0023] a=(a x a y )=(min(x T ,x S ,x U ,x G )max(y T ,y S ,y U ,y G )),

[0024] b = (b x b y )=(max(x T ,x S ,x U ,x G )max(y T ,y S ,y U ,y G )),

[0025] c = (c x c y )=(min(x T ,xS ,x U ,x G min(y) T ,y S ,y U ,y G )),

[0026] d=(d x d y )=(max(x T ,x S ,x U ,x G min(y) T ,y S ,y U ,y G ));

[0027] Expand the coordinate matrix to

[0028] as well as

[0029] With coordinate matrix Using the vertex as the vertices, extract the four video channels: T, S, U, and G.

[0030] Furthermore, in the above technical solution, the abnormal working condition annotation and abnormal coordinate point extraction of the abnormal working condition sample video, and the training of the second convolutional neural network model include:

[0031] Extract the abnormal coordinates A from the T, S, U, and G channels of the abnormal operating condition sample video. T (x T ,y T A) S (x S ,y S A) U (x U ,y U ) and A G (x G ,y G );

[0032] Compare A T A S A U and A G The coordinate values ​​are used to obtain the coordinate matrix.

[0033] Expanding coordinate matrix A by σ yields coordinate matrix A. as well as

[0034] Using coordinate matrix A' as the vertex, the T, S, U, and G channels of the abnormal working condition sample video are extracted as input, and the type of abnormal working condition is used as the encoded output.

[0035] Furthermore, in the above technical solution, abnormal coordinate points are extracted from the real-time monitoring video, and input into the abnormal operating condition recognition model and the target equipment recognition model to obtain the abnormal recognition results, including:

[0036] Extracting the abnormal coordinates B of the four channels (T, S, U, and G) of the real-time monitoring video. T (x T ,y T B S (x S ,y S B U (x U ,y U ) and B G (x G ,y G );

[0037] Compare B T B S B U and B G The coordinate values ​​are used to obtain the coordinate matrix.

[0038] Expanding the coordinate matrix B by σ yields the coordinate matrix.

[0039]

[0040] Using coordinate matrix B' as the vertex, extract the T, S, U, and G channels of the real-time monitoring video, input them into the abnormal operating condition recognition model, and obtain the abnormal operating condition type; and

[0041] Using coordinate matrix B' as the vertex, extract the R, G, and B channels of the real-time monitoring video, input them into the target device recognition model, and obtain the type of device that has experienced an anomaly.

[0042] Furthermore, in the above technical solution, the anomaly identification results include the type of abnormal operating condition, the location of the anomaly, and the type of equipment that experienced the anomaly.

[0043] Furthermore, the anomaly identification method of multiphysics field fusion in the above technical solution also includes the step of issuing an alarm message based on the anomaly identification result.

[0044] According to a second aspect of the present invention, the present invention provides a multi-physics field fusion anomaly identification system, comprising: a multi-physics field video acquisition unit for acquiring sample videos of monitored objects, sample videos of abnormal operating conditions, and real-time monitoring videos; a data processing unit for annotating the sample videos of monitored objects with target equipment and extracting them sequentially and individually, annotating the sample videos of abnormal operating conditions with abnormal operating conditions and extracting abnormal coordinate points, and extracting abnormal coordinate points from the real-time monitoring videos; a model training unit for training a first convolutional neural network model and a second convolutional neural network model based on the results of the data processing unit to establish a target equipment identification model and an abnormal operating condition identification model; an anomaly identification unit for performing anomaly identification on the real-time monitoring videos based on the target equipment identification model and the abnormal operating condition identification model; and an output unit for outputting the anomaly identification results.

[0045] Furthermore, in the above technical solution, the multiphysics video acquisition unit includes: an infrared temperature measurement imaging module, an infrared gas imaging module, an ultraviolet light imaging module, a visible light camera module, and a sound field acquisition module.

[0046] Furthermore, in the above technical solution, the multi-physics field fusion anomaly identification system also includes an alarm unit, which is used to issue alarm information based on the anomaly identification results.

[0047] According to a third aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform a multiphysics fusion anomaly identification method as described in any of the above technical solutions.

[0048] According to a fourth aspect of the present invention, the present invention provides a non-transitory computer-readable storage medium storing computer-executable instructions for causing a computer to perform an anomaly identification method of multiphysics fusion as described in any of the above technical solutions.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. This invention fuses multi-physics field videos and, based on a comprehensive judgment of multiple physical quantities, increases the number of identifiable anomaly types and improves the accuracy of anomaly identification.

[0051] 2. This invention uses a dual convolutional neural network model to automatically and quickly identify the types, locations, and types of equipment experiencing various abnormal operating conditions.

[0052] 3. Based on the anomaly identification results, this invention issues alarm information, enabling relevant personnel to handle the situation promptly, thereby preventing serious accidents and ensuring production safety.

[0053] 4. This invention integrates various acoustic and optical surface monitoring devices that previously required manual confirmation. It fuses and analyzes the changes in multiple physical quantities, replacing the feature point extraction and localization steps in ordinary image processing algorithms with abnormal coordinate points in the imaging monitoring screens of multiple physical quantities. This significantly improves the efficiency of model operation in training and actual testing, enabling automatic identification and alarm for abnormal operating conditions such as leaks, fires, vibration noise, and partial discharge. It can also automatically identify the type of equipment malfunctioning. Alarm information containing the abnormal state and location is sent to management personnel, achieving intelligent, large-scale anomaly monitoring and identification.

[0054] 5. This invention enables imaging monitoring equipment to have autonomous analysis, identification, and alarm capabilities, and can be used in scenarios such as fixed on-site monitoring or mobile robot monitoring, greatly expanding the application scope of imaging monitoring equipment.

[0055] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, and to make the above and other objects, technical features and advantages of the present invention easier to understand, one or more preferred embodiments are listed below and described in detail with reference to the accompanying drawings. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating an anomaly identification method based on multiphysics field fusion according to an embodiment of the present invention.

[0057] Figure 2 This is a schematic diagram of the structure of an anomaly identification system based on multiphysics field fusion according to an embodiment of the present invention.

[0058] Figure 3 This is a schematic diagram of the process for extracting abnormal coordinate points from abnormal working condition sample videos according to an embodiment of the present invention.

[0059] Figure 4 This is a schematic diagram of the hardware structure of an electronic device that performs an anomaly identification method for multiphysics fusion according to an embodiment of the present invention. Detailed Implementation

[0060] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0061] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0062] In this document, for ease of description, spatial relative terms such as “below,” “under,” “down,” “above,” “above,” “up,” etc., are used to describe the relationship of one element or feature to another element or feature in the accompanying drawings. It should be understood that spatial relative terms are intended to encompass different orientations of an object in use or operation, in addition to those depicted in the figures. For example, if an object in the figure is flipped, an element described as “below” or “under” another element or feature would be oriented “above” that element or feature. Thus, the exemplary term “below” can encompass both the downward and upward orientations. An object may also have other orientations (rotated 90 degrees or other orientations), and the spatial relative terms used herein should be interpreted accordingly.

[0063] In this document, the terms "first," "second," etc., are used to distinguish two different elements or parts, and are not used to define specific positions or relative relationships. In other words, in some embodiments, the terms "first," "second," etc., can also be used interchangeably.

[0064] like Figure 1 As shown, the flow of the multiphysics fusion anomaly identification method according to a specific embodiment of the present invention is as follows:

[0065] S110 acquires sample videos of monitored objects, sample videos of abnormal operating conditions, and real-time monitoring videos.

[0066] The monitored object sample video is a visible light video, the abnormal operating condition sample video is a multi-physics field video, and the real-time monitoring video includes both visible light video and multi-physics field video. Further, in one or more exemplary embodiments of the present invention, the multi-physics field video may include infrared thermography video, infrared gas imaging video, ultraviolet light distribution video, and sound field cloud image video. Exemplarily, the monitored object sample video can be captured using a visible light camera, and the abnormal operating condition sample video can be formed by combining video data simultaneously collected from the simulated experimental device by four types of devices: an infrared thermography imager, an infrared gas imager, an ultraviolet imager, and a sound field imager. The infrared thermography imager outputs infrared thermography video; the video output by the infrared gas imager needs to use algorithms such as inter-frame difference to extract the pixels where the gas cloud target is located and subtract other backgrounds to form an infrared gas imaging video; the ultraviolet imager outputs ultraviolet spot monitoring results and subtracts other backgrounds to form an ultraviolet light distribution video; the sound field imager uses a controllable beamforming algorithm based on maximum power output to obtain a color contour image of the sound field distribution after beamforming calculation, which is then made transparent to form a sound field cloud image video.

[0067] S120 standardizes the sample videos of monitored objects, sample videos of abnormal operating conditions, and real-time monitoring videos.

[0068] Furthermore, in one or more exemplary embodiments of the present invention, video standardization can be achieved using distortion correction, image cropping, or other methods to make the aspect ratio, field of view, and target object size and shape of the multiphysics video the same or as similar as possible, that is, to ensure that the same target shape has the same size, aspect ratio, and shape in the multiphysics video image. Video standardization also includes resolution standardization. For example, the median resolution of the multiphysics video image is taken as the standard resolution. Videos with resolutions lower than the standard resolution are upscaled to the standard resolution using an algorithm, and videos with resolutions higher than the median resolution are downscaled to the standard resolution using an algorithm.

[0069] S130 labels standardized monitoring object sample videos and abnormal operating condition sample videos.

[0070] Furthermore, in one or more exemplary embodiments of the present invention, regions of target equipment such as pumps, valves, transformers, pipelines, flanges, steel structures, pressure vessels, storage tanks, and towers are selected in standardized sample video images of monitored objects to obtain the coordinate range of the corresponding type of target equipment in the sample video images, and the equipment type is labeled. Coordinate ranges of abnormal operating conditions such as leakage, vibration, noise, flame, and partial discharge are selected in standardized abnormal operating condition sample videos, and the type is labeled.

[0071] S140 model training was used to establish target equipment identification models and abnormal operating condition identification models.

[0072] S141 extracts the monitored object sample video sequentially and individually according to the coordinate range of the target device, and divides it into R, G, and B three-channel video as input to train the first convolutional neural network model. The type of the target device is used as the encoded output to establish a target device recognition model. The target device recognition model enables machine recognition of the type of the monitored target device.

[0073] For example, the three-channel video uses the RGB color mode with an intensity range of 0 to 255.

[0074] S142 extracts abnormal coordinate points from abnormal working condition sample videos, trains a second convolutional neural network model, and establishes an abnormal working condition recognition model.

[0075] Furthermore, in one or more exemplary embodiments of the present invention, infrared thermography video is used as the first channel (T channel) video, and sound field cloud image video (S channel), ultraviolet light distribution video (U channel), and infrared gas imaging video (G channel) are used as the second, third, and fourth channel videos, respectively. Grayscale transformation is performed on the four channels (T, S, U, and G channels) to proportionally scale the detectable temperature range of the infrared thermography imager to a value range of 0–255; proportionally scale the grayscale value range of the infrared gas imager to a value range of 0–255; proportionally scale the detectable ultraviolet light intensity range of the ultraviolet imager to a value range of 0–255; and proportionally scale the detectable sound intensity range of the sound field imager to a value range of 0–255. The four-channel color video images are then converted into grayscale images.

[0076] Establish an x0y coordinate system using the 2D video image, and extract the abnormal coordinate points of the four video channels T, S, U, and G: If there is a temperature anomaly, extract the coordinates of the center point of the temperature anomaly area in the T channel video (x0y). T ,y T If there is noise anomaly, extract the coordinates (x, y) of the center point of the noise region in the S-channel video. S ,y S If there is an ultraviolet light anomaly, extract the coordinates (x, y) of the center point of the ultraviolet light source in the U-channel video. U ,y U ); and if there is a gas anomaly, extract the coordinates (x, y) of the center point of the gas cloud region in the G channel video. G ,y G For example, areas with abnormal temperatures can be identified by establishing a normal operating temperature database and comparing the monitoring results with the normal operating temperature database.

[0077] By comparing the coordinate values, a coordinate matrix is ​​obtained. in,

[0078] a=(a x a y )=(min(x T ,x S ,x U ,x G )max(y T ,y S ,y U ,y G )),

[0079] b = (b x b y )=(max(x T ,x S ,x U ,x G )max(y T ,yS ,y U ,y G )),

[0080] c = (c x c y )=(min(x T ,x S ,x U ,x G min(y) T ,y S ,y U ,y G )),

[0081] d=(d x d y )=(max(x T ,x S ,x U ,x G min(y) T ,y S ,y U ,y G ));

[0082] Expand the coordinate matrix to

[0083] as well as

[0084] With coordinate matrix Using the vertex as the vertices, extract the four video channels: T, S, U, and G.

[0085] The above method was used to extract the abnormal coordinate point A from the four channels (T, S, U, and G) of the abnormal operating condition sample video. T (x T ,y T A) S (x S ,y S A) U (x U ,y U ) and A G (x G ,y G Compare A T A S A U and A G The coordinate values ​​are used to obtain the coordinate matrix. Expanding coordinate matrix A by σ yields coordinate matrix A. as well as

[0086] Using coordinate matrix A' as the vertex, the T, S, U, and G channels of the abnormal working condition sample video are extracted as input, and the type of abnormal working condition is used as the encoded output to train the second convolutional neural network model, thus establishing an abnormal working condition recognition model.

[0087] The S150 extracts abnormal coordinate points from real-time monitoring videos, inputs them into the abnormal operating condition recognition model and the target equipment recognition model, and obtains the abnormal recognition results.

[0088] Furthermore, in one or more exemplary embodiments of the present invention, the established abnormal operating condition identification model and target equipment identification model are placed into the monitoring host computer. At the start of the actual testing phase, it is first determined whether any abnormal conditions appear in the T, S, U, and G channels of the real-time monitoring video. If one or more abnormal conditions (temperature abnormality, noise abnormality, ultraviolet light abnormality, and / or gas abnormality, etc.) appear, the abnormal coordinate point B of the T, S, U, and G channels of the real-time monitoring video is extracted. T (x T ,y T B S (x S ,y S B U (x U ,y U ) and B G (x G ,y G Compare B T B S B U and B G The coordinate values ​​are used to obtain the coordinate matrix. Expand the selected area of ​​coordinate matrix B by σ to obtain the coordinate matrix.

[0089] Using coordinate matrix B' as the vertex, extract the T, S, U, and G channels of the real-time monitoring video, input them into the abnormal operating condition recognition model, and obtain the abnormal operating condition type; and using coordinate matrix B' as the vertex, extract the R, G, and B channels of the visible light video from the real-time monitoring video, input them into the target device recognition model, and obtain the device type that has an abnormality.

[0090] Based on the anomaly identification results, S160 issues an alarm message.

[0091] Furthermore, in one or more exemplary embodiments of the present invention, based on the type of equipment and the type of abnormal operating condition in the anomaly identification results, an alarm message is sent to the safety management personnel, such as "There may be a gas leak and fire in pump A, please check immediately!"

[0092] like Figure 2As shown, the multi-physics fusion anomaly identification system according to a specific embodiment of the present invention includes: a multi-physics video acquisition unit 10, used to acquire sample videos of monitored objects, sample videos of abnormal operating conditions, and real-time monitoring videos; a data processing unit 20, used to annotate the sample videos of monitored objects with target equipment and extract them sequentially and individually, annotate the sample videos of abnormal operating conditions with abnormal operating conditions and extract abnormal coordinate points, and extract abnormal coordinate points from the real-time monitoring videos; a model training unit 30, used to train a first convolutional neural network model and a second convolutional neural network model based on the results of the data processing unit to establish a target equipment identification model and an abnormal operating condition identification model; an anomaly identification unit 40, used to perform anomaly identification on the real-time monitoring videos based on the target equipment identification model and the abnormal operating condition identification model; and an output unit 50, used to output the anomaly identification results.

[0093] Furthermore, in one or more exemplary embodiments of the present invention, the multiphysics video acquisition unit 10 may include: an infrared temperature measurement imaging module, an infrared gas imaging module, an ultraviolet light imaging module, a visible light camera module, and a sound field acquisition module.

[0094] Furthermore, in one or more exemplary embodiments of the present invention, the multiphysics fusion anomaly identification system further includes an alarm unit (not shown in the figure), which is used to issue alarm information based on the anomaly identification result.

[0095] The following describes in more detail the multiphysics fusion anomaly identification method, system, electronic device, and storage medium of the present invention through specific embodiments. It should be understood that the embodiments are merely exemplary and the present invention is not limited thereto.

[0096] Example 1

[0097] refer to Figure 3 As shown, this embodiment illustrates the process of extracting abnormal coordinate points from abnormal working condition sample videos.

[0098] Based on the abnormal state, extract the coordinates A of the center point of the temperature anomaly region in the T-channel video. T (x T ,y T Extract the coordinates A of the center point of the noise region in the S-channel video. S (x S ,y S Extract the coordinates A of the center point of the ultraviolet light source in the U-channel video. U (x U ,y U ); and extract the coordinates A of the center point of the gas cloud region in the G channel video. G (x G ,y G ).

[0099] Compare A T A S A U and A G The coordinate values ​​are used to obtain the coordinate matrix. in,

[0100] a=(a x a y )=(min(x T ,x S ,x U ,x G )max(y T ,y S ,y U ,y G )),

[0101] b = (b x b y )=(max(x T ,x S ,x U ,x G )max(y T ,y S ,y U ,y G )),

[0102] c = (c x c y )=(min(x T ,x S ,x U ,x G min(y) T ,y S ,y U ,y G )),

[0103] d=(d x d y )=(max(x T ,x S ,x U ,x G min(y) T ,y S ,y U ,y G )).

[0104] Expanding coordinate matrix A by σ yields coordinate matrix A. as well as

[0105] Using coordinate matrix A' as the vertex, extract the T, S, U, and G channels of the abnormal working condition sample video.

[0106] Example 2

[0107] This embodiment illustrates the multi-physics video acquisition unit in the multi-physics anomaly identification system according to the present invention. The multi-physics video acquisition unit includes a Dali D843NT infrared thermography imaging module, a FLIR GF77a infrared gas imaging module, a Lipuzheng LPZ-UV2 ultraviolet imaging module, and a Sony IMX317 visible light camera module. It uses 32 Knowles 3526-MEMS microphones arranged in a helical array to form a microphone array as the sound field acquisition module. The 32-channel audio signal uses beamforming technology based on maximum power output to obtain a color contour map of the sound field, forming a real-time sound field cloud image video.

[0108] Example 3

[0109] This embodiment provides a non-transitory (non-volatile) computer storage medium that stores computer-executable instructions that can execute the methods in any of the above method embodiments and achieve the same technical effect.

[0110] Example 4

[0111] This embodiment provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform the methods described above and achieve the same technical effects.

[0112] Example 5

[0113] Figure 4 This is a schematic diagram of the hardware structure of an electronic device that performs the multiphysics fusion anomaly detection method according to this embodiment. The device includes one or more processors 610 and a memory 620. Taking one processor 610 as an example, the device may also include an input device 630 and an output device 640.

[0114] The processor 610, memory 620, input device 630, and output device 640 can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0115] The memory 620, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 610 executes various functional applications and data processing of the electronic device by running the non-transitory software programs, instructions, and modules stored in the memory 620, thereby implementing the processing method of the above-described method embodiments.

[0116] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 620 may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories may be connected to the processing device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0117] Input device 630 can receive input digital or character information and generate signal input. Output device 640 may include display devices such as a display screen.

[0118] One or more modules are stored in memory 620 and, when executed by one or more processors 610, execute:

[0119] Acquire sample videos of the monitored objects, sample videos of abnormal operating conditions, and real-time monitoring videos. The sample videos of the monitored objects are visible light videos, while the sample videos of abnormal operating conditions and real-time monitoring videos are multi-physics field videos.

[0120] The target devices in the monitored sample videos are labeled and extracted one by one in sequence. The first convolutional neural network model is trained to establish a target device recognition model.

[0121] Abnormal working condition sample videos are labeled with abnormal working conditions and abnormal coordinate points are extracted. A second convolutional neural network model is trained to establish an abnormal working condition recognition model.

[0122] Anomaly coordinates are extracted from real-time monitoring video, and the results are obtained by inputting the abnormal operating condition recognition model and the target equipment recognition model.

[0123] The above-described product can execute the methods provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in other embodiments of the present invention.

[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general-purpose hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0126] The foregoing description of specific exemplary embodiments of the present invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. Any simple modifications, equivalent changes, and alterations made to the foregoing exemplary embodiments should fall within the scope of protection of the present invention.

Claims

1. An anomaly identification method based on multiphysics field fusion, characterized in that, Includes the following steps: The system acquires sample videos of monitored objects, sample videos of abnormal operating conditions, and real-time monitoring videos. The sample videos of monitored objects are visible light videos, the sample videos of abnormal operating conditions are multi-physics field videos, and the real-time monitoring videos include visible light videos and multi-physics field videos. The multi-physics field videos include infrared thermometry videos, infrared gas imaging videos, ultraviolet light distribution videos, and sound field cloud map videos. The target devices in the monitored sample videos are labeled and extracted one by one in sequence. The first convolutional neural network model is trained to establish a target device recognition model. Abnormal working condition sample videos are labeled with abnormal working conditions and abnormal coordinate points are extracted. A second convolutional neural network model is trained to establish an abnormal working condition recognition model. Extract abnormal coordinate points from real-time monitoring video, input them into the abnormal operating condition recognition model and the target equipment recognition model, and obtain the abnormal recognition results; Anomaly coordinate point extraction includes: Infrared thermometry video, sound field cloud map video, ultraviolet light distribution video, and infrared gas imaging video are respectively used as T, S, U, and G four-channel videos; Perform grayscale transformation on the four channels of video: T, S, U, and G.

2. The anomaly identification method based on multiphysics fusion according to claim 1, characterized in that, It also includes the step of standardizing the sample videos of the monitored objects, the sample videos of abnormal operating conditions, and the real-time monitoring videos.

3. The anomaly identification method based on multiphysics fusion according to claim 2, characterized in that, Standardization includes resolution standardization, distortion correction, and image cropping.

4. The anomaly identification method based on multiphysics fusion according to claim 1, characterized in that, The types of target equipment include pumps, valves, transformers, pipelines, flanges, steel structures, pressure vessels, storage tanks, and towers; the types of abnormal operating conditions include leakage, vibration, noise, flame, and partial discharge.

5. The anomaly identification method based on multiphysics fusion according to claim 1, characterized in that, When training the first convolutional neural network model, the sample videos of the monitored objects are extracted one by one according to the coordinate range of the target device, and divided into three channels of video (R, G, B) as input, and the type of the target device is used as the encoded output.

6. The anomaly identification method based on multiphysics fusion according to claim 1, characterized in that, Extracting outlier coordinates from the T, S, U, and G channels of the video: If there is a temperature anomaly, extract the coordinates of the center point of the temperature anomaly area in the T-channel video. ; If there is noise anomaly, extract the coordinates of the center point of the noise region in the S-channel video. ; If there is an ultraviolet light anomaly, extract the coordinates of the center point of the ultraviolet light source from the U-channel video. ; as well as If there is a gas anomaly, extract the coordinates of the center point of the gas cloud region in the G channel video. ; By comparing the coordinate values, a coordinate matrix is ​​obtained. ,in, ， ， ， ； Expand the coordinate matrix to ； as well as With coordinate matrix Using the vertex as the vertices, extract the four video channels: T, S, U, and G.

7. The anomaly identification method based on multiphysics fusion according to claim 6, characterized in that, The abnormal operating condition sample videos are labeled with abnormal operating conditions and abnormal coordinate points are extracted. The training of the second convolutional neural network model includes: Extracting abnormal coordinates from the T, S, U, and G channels of abnormal operating condition video samples. , , and ; Compare , , and The coordinate values ​​are used to obtain the coordinate matrix. ; Expand the coordinate matrix A Obtain the coordinate matrix ;as well as With coordinate matrix Using the vertex as the input, the T, S, U, and G channels of the abnormal working condition sample video are extracted as input, and the type of abnormal working condition is used as the encoded output.

8. The anomaly identification method based on multiphysics fusion according to claim 7, characterized in that, Anomaly coordinates are extracted from real-time monitoring video. These are then input into anomaly identification models and target equipment identification models to obtain anomaly identification results, including: Extracting abnormal coordinates from the T, S, U, and G channels of real-time surveillance video. , , and ; Compare , , and The coordinate values ​​are used to obtain the coordinate matrix. ; Expand the coordinate matrix B Obtain the coordinate matrix ； With coordinate matrix Using the vertex as the endpoint, extract the T, S, U, and G channels of the real-time monitoring video, input them into the abnormal operating condition recognition model, and obtain the abnormal operating condition type; and With coordinate matrix Using the vertex as the vertices, extract the R, G, and B channels of the real-time monitoring video, input them into the target device identification model, and obtain the type of device that has experienced an anomaly.

9. The anomaly identification method based on multiphysics fusion according to claim 1, characterized in that, The anomaly identification results include the type of abnormal operating condition, the location of the anomaly, and the type of equipment that experienced the anomaly.

10. The anomaly identification method based on multiphysics fusion according to claim 9, characterized in that, It also includes the following steps: An alarm message is issued based on the anomaly identification results.

11. An anomaly detection system based on multiphysics field fusion, characterized in that, The method described in any one of claims 1 to 10 includes: Multiphysics video acquisition unit, which is used to acquire sample videos of monitored objects, sample videos of abnormal operating conditions, and real-time monitoring videos; The data processing unit is used to annotate the target equipment in the sample video of the monitored object and extract it separately in turn, to annotate the abnormal working condition sample video and extract the abnormal coordinate points, and to extract the abnormal coordinate points in the real-time monitoring video. The model training unit is used to train the first convolutional neural network model and the second convolutional neural network model based on the results of the data processing unit, so as to establish the target equipment recognition model and the abnormal working condition recognition model. Anomaly detection unit, which performs anomaly detection on real-time monitoring video based on target device detection model and abnormal operating condition detection model; and The output unit is used to output the anomaly identification results.

12. The anomaly detection system based on multiphysics field fusion according to claim 11, characterized in that, The multiphysics video acquisition unit includes: an infrared temperature measurement imaging module, an infrared gas imaging module, an ultraviolet light imaging module, a visible light camera module, and a sound field acquisition module.

13. The anomaly detection system based on multiphysics fusion according to claim 11, characterized in that, Also includes: An alarm unit is used to issue alarm information based on the anomaly identification results.

14. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, causes the at least one processor to perform the multiphysics fusion anomaly identification method as described in any one of claims 1 to 10.

15. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions for causing the computer to perform the anomaly identification method of multiphysics fusion as described in any one of claims 1 to 10.

Images