Infrared image fault diagnosis method, infrared imaging device, medium and program product
By identifying and configuring the object categories and contours in the infrared image data, and automatically drawing the contours of the parts to be tested in combination with diagnostic rules, the problem of accurate drawing and complex object recognition of infrared thermal imagers in troubleshooting is solved, and detection efficiency and flexibility are improved.
Patent Information
- Application Number
- CN202510422878.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-07
- Publication Date
- 2025-07-04
AI Technical Summary
Existing infrared thermal cameras cannot accurately draw the contour of the target area under test in fault diagnosis, resulting in large environmental interference and time-consuming drawing. They also require high industry knowledge of users, are difficult to identify complex targets, and are difficult to implement diagnostic rules.
By obtaining infrared image data, identifying the object categories and contours of the part, configuring the part categories and diagnostic rules, automatically identifying and drawing the outlines of the part to be tested, and troubleshooting is performed in combination with ledger associations, reducing the difficulty of identification and simplifying steps.
It improves the flexibility and detection efficiency of target recognition, reduces the knowledge reserve requirements for users, simplifies workflow, and reduces environmental interference.
Smart Images
Figure CN120259271A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of infrared image, and particularly to an infrared image fault diagnosis method, an infrared imaging device, a computer-readable storage medium and a computer program product. Background Art
[0002] Currently, infrared thermal imagers are mainly applied to fault diagnosis. However, during the application process, there are often problems that the regional contour of the measured target cannot be accurately drawn, either resulting in great interference from the surrounding environment to the temperature detection of the target, or requiring the user to spend a lot of time carefully drawing the measured area of the measured target; in addition, due to the lack of intelligent functions for assistance, the requirement for the industry knowledge reserve of the user is relatively high. Therefore, based on the above situation, the need for intelligent fault diagnosis has emerged. In addition, during the implementation process of the intelligent diagnosis solution, for complex targets, the recognition difficulty is very high and the accuracy rate is low; and during the rule formulation process, the diagnosis rules for different parts of complex targets are different, and not all areas need to be diagnosed. Therefore, if the target is regarded as a single entity to formulate rules, the implementation difficulty is relatively large. Summary of the Invention
[0003] In order to solve the existing technical problems, the present invention provides an infrared image fault diagnosis method, an infrared imaging device, a computer-readable storage medium and a computer program product, which can detect the parts to be measured that need attention, reduce the difficulty of the algorithm in identifying the target, and improve the detection efficiency.
[0004] In the first aspect, an infrared image fault diagnosis method is provided, including: acquiring infrared image data collected for a target to be measured; acquiring the part category corresponding to the target to be measured; performing target recognition on the infrared image data to determine the part object category and part object contour of the part object in the infrared image data; determining the target part category based on the part category and the part object category, and determining the target part contour of the part to be measured based on the target part category and the part object contour; acquiring a diagnosis rule matching the target part category where the part to be measured is located, and obtaining a fault diagnosis result of the target to be measured based on the diagnosis rule and the target part contour of the part to be measured.
[0005] In the second aspect, an infrared imaging device is provided, including a memory, a processor and an infrared image acquisition device; the infrared image acquisition device acquires infrared image data; a computer program is stored in the processor, and the processor is configured to execute the computer program to implement the infrared image fault diagnosis method according to any item in the first aspect of the present application.
[0006] In a third aspect, an infrared image fault diagnosis system is provided, including the infrared imaging device and the terminal device as described in any item of the second aspect, where the terminal device is configured to configure the part categories corresponding to the target to be measured and the diagnosis rules corresponding to each of the part categories.
[0007] In a fourth aspect, a computer program product is provided, including a computer program, which when executed by a processor, implements an infrared image fault diagnosis method as described in any item of the first aspect of the present application.
[0008] In a fifth aspect, a computer-readable storage medium is provided, including a computer program, which when executed by a processor, implements an infrared image fault diagnosis method as described in any item of the first aspect of the present application.
[0009] In the present application, the part categories corresponding to the target to be measured and the diagnosis rules corresponding to the part categories are pre-configured. By obtaining the infrared image data of the target to be measured, the part object category and the part object contour of the part object of the target to be measured are identified from the infrared image data, and the part to be measured that needs attention is detected, reducing the difficulty of the algorithm in identifying the target. Based on the configured part categories and the identified part object categories, the target part category that finally needs to be detected is determined. Based on the target part category, the target part contour is determined. Based on the target part contour, the temperature data of the corresponding part to be measured is determined. Based on the part category corresponding to the target part contour, the diagnosis rule corresponding to the corresponding part to be measured is determined, and the fault diagnosis results corresponding to each target part contour are obtained. According to the fault diagnosis results corresponding to each target part contour, the fault diagnosis result of the target to be measured can be obtained. By associating with the account book, when diagnosing, the fault diagnosis result of the target to be measured is determined according to the fault diagnosis results of each part, increasing the flexibility of target recognition; by integrating the diagnosis rules of the target, the knowledge reserve requirements for the users are reduced, the steps of the staff are simplified, the detection efficiency is improved, and convenience is provided for the users. Description of the Drawings
[0010] Figure 1 It is an application environment diagram of an infrared image fault diagnosis method in an embodiment;
[0011] Figure 2 It is a flowchart of an infrared image fault diagnosis method in an embodiment;
[0012] Figure 3 It is a display schematic diagram of an image fault diagnosis result in an embodiment;
[0013] Figure 4 It is a schematic diagram of an infrared image fault diagnosis device in an embodiment;
[0014] Figure 5Schematic diagram of an infrared imaging device in an embodiment. Detailed implementation manners
[0015] The technical solution of the present invention will be further elaborated in detail below in conjunction with the accompanying drawings of the specification and specific embodiments.
[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this invention belongs. The terms used in the specification of this invention herein are only for the purpose of describing specific embodiments, and are not intended to limit the protection scope of this invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.
[0017] In the following description, the expression "some embodiments" describes a subset of all possible embodiments. However, it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.
[0018] Please refer to Figure 1 , which is a schematic diagram of an optional application scenario of the infrared image fault diagnosis method provided by the embodiment of this application. The infrared image fault diagnosis method is applied to an infrared imaging device. The infrared imaging device can be various devices for collecting infrared image data such as an infrared thermal imager, a thermal imaging temperature measuring instrument, or other intelligent devices integrated with devices for collecting infrared image data such as an infrared thermal imager and a thermal imaging temperature measuring instrument and having communication and storage functions. Among them, the infrared image fault diagnosis method may include the following processes: S1, obtaining infrared image data collected for a target to be measured; S2, obtaining the part category corresponding to the target to be measured; S3, performing target recognition on the infrared image data to determine the part object category and part object contour of the part object in the infrared image data; S4, determining the target part category based on the part category and the part object category, and determining the target part contour of the part to be measured based on the target part category and the part object contour; S5, obtaining a diagnostic rule matching the target part category where the part to be measured is located, and obtaining a fault diagnosis result of the target to be measured based on the diagnostic rule and the target part contour of the part to be measured.
[0019] In some optional embodiments, the infrared imaging device may refer to an infrared image fault diagnosis system to which the infrared image fault diagnosis method is applied. The infrared image fault diagnosis system includes a plurality of physically separated hardware devices, such as an infrared image acquisition device and a data processing device communicatively connected to the infrared image acquisition device. The infrared image acquisition device is used to execute the S1 process in the infrared image fault diagnosis method, and the data processing device is used to execute the S2-S5 processes in the infrared image fault diagnosis method.
[0020] In some other alternative embodiments, the infrared imaging device includes an infrared thermal imager and a terminal device communicatively connected to the infrared thermal imager. The terminal device is configured to execute the process of S3 in the infrared image fault diagnosis method, and the infrared thermal imager is configured to execute the processes of S1, S2, S4, and S5 in the infrared image fault diagnosis method.
[0021] The infrared image acquisition device can be a combination of one or more sensors. The infrared image acquisition device can be a monocular vision sensor or a multiocular vision sensor. For example, it can be a combination of one or more sensors such as a thermal imaging sensor, a visible light image sensor, a millimeter wave sensor, a lidar sensor, an infrared thermal imaging sensor, a depth sensor, a back-illuminated CMOS sensor, an electron multiplying CCD sensor, a scientific-grade CMOS sensor, an InGaAs sensor, a microchannel plate sensor, and a quantum dot sensor.
[0022] The infrared imaging device may further include other sensor modules, including but not limited to an environmental perception sensor and a motion attitude sensor. The environmental perception sensor includes but is not limited to one or more combinations of the following sensors: a brightness sensor, a temperature sensor, a haze sensor, and other environmental sensors. The motion attitude sensor includes but is not limited to one or more combinations of the following: an inertial sensor (Inertial Measurement Unit, IMU), a speed sensor, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, a rotation vector sensor, a steering wheel angle sensor, a level sensor, an inclination sensor, a vibration sensor, a displacement sensor, and a gravity sensor.
[0023] The infrared imaging device may further include a display terminal for displaying fault diagnosis data and data required to configure and implement the infrared image fault diagnosis method provided in the embodiments of the present application.
[0024] Please refer to Figure 2 , which is a flowchart of an infrared image fault diagnosis method provided in an embodiment of the present application. An infrared image fault diagnosis method is applied to an infrared imaging device. The infrared image fault diagnosis method includes the following steps:
[0025] S1. Obtain infrared image data collected for a target to be measured.
[0026] In this embodiment, the infrared image data may refer to infrared imaging pictures or infrared video frame data collected by a device with the function of collecting infrared image data; it may also refer to infrared imaging pictures or infrared video frame data obtained from a device with the function of collecting infrared image data and sent by it. The infrared image data may be collected in real time by a device with the function of collecting infrared image data, or may be pre-collected and stored. The infrared image data may correspondingly refer to multiple infrared imaging pictures collected in real time, or called from an image database, or obtained from other devices through communication, or multiple infrared image frames extracted from infrared video data. The target to be measured may be one or more of a certain type of target.
[0027] It should be noted that the infrared image is generated based on the infrared radiation (thermal radiation) emitted or reflected by an object. Thus, the infrared image actually contains both image information and temperature information. In an optional specific example, the infrared image data includes temperature data and image data, and can be represented by a temperature data set T mn and an image data set Y mn to represent.
[0028] S2. Obtain the part category corresponding to the target to be measured.
[0029] In this embodiment, the target to be measured represents the same type of target to be detected currently, and the shapes of this type of target are roughly the same. The target to be measured may be a target selected by the user through a user interface, or may be the target currently detected according to the task progress of the inspection task in the ledger. A target to be measured can be disassembled into multiple parts, but only the parts to be measured that need to be detected need to be concerned. The parts to be measured are the parts that need to be detected. In this way, it is not necessary to detect the entire target to be measured. Subsequently, only the parts to be measured that need to be detected need to be detected. For example, a target to be measured can be disassembled into A1, A2, and A3, and the user only concerns part A3, then A3 is the part to be measured. For one or more targets to be measured included in the inspection task, multiple targets to be measured can be disassembled in advance to obtain all parts to be measured, and these all parts to be measured are classified according to the shape category to obtain the part category. The part category may be one or more categories, such as including but not limited to one or more of the following combinations: rectangle, circle, square, shapes combined by multiple geometric shapes, etc. In this way, for a target to be measured, it corresponds to one or more part categories, and each part category corresponds to a diagnosis rule. For different part categories, they may correspond to the same or different diagnosis rules. For example, the target to be measured A has part categories C1 and C2, and the target to be measured B has part categories C1 and C3. For different targets, the target to be measured A and the target to be measured B may have the same part category, and the diagnosis rule corresponding to part category C1 in the target to be measured B may be different from or the same as the diagnosis rule corresponding to part category C1 in the target to be measured A.
[0030] In this embodiment, the set of targets to be measured forms a set of targets to be measured G_List, and the parts to be measured are classified to obtain a set of part categories TP A (A is the total number of categories). And record the part category set TP of the target G (G ∈ G_List) to be measured G (TP G ∈ TP A , TP G .count > 0). The process of disassembling each target to be measured above is similar to the process of modularizing a target
[0031] S3. Perform target recognition on the infrared image data to determine the part object category and part object contour of the part object in the infrared image data
[0032] In this embodiment, the part object indicates one or more local parts in the target to be measured. The part object category indicates the part category corresponding to the recognized part target, and the part object contour indicates the contour edge of the recognized part object. Among them, the part object can be one or more parts in the target to be measured. For example, the part object refers to parts S01, S02, and S03 corresponding to device A. Perform target recognition on the infrared image data to determine parts S01, S02, and S03 of device A included in the infrared image, and determine the respective edge contours and corresponding part categories of the recognized parts S01, S02, and S03
[0033] S4. Determine the target part category based on the part category and the part object category, and determine the target part contour of the part to be measured based on the target part category and the part object contour
[0034] In this embodiment, the part category is the part category corresponding to the target to be measured pre-configured by the user, and the part object category is the part category recognized based on the infrared image data. Some part categories that may be recognized are not in the part categories that need to be detected. Therefore, it is necessary to determine the final part category to be detected based on the part category and the part object category, that is, the target part category. Among them, the target part category can be one or more, and each target part category can include one or more parts. For example, under the target part category A1 of the target to be measured A, there are parts S01 and S02, and under the part category A2, there is part S03
[0035] The target part contour indicates the final contour corresponding to the part to be measured for extracting the target temperature data. One part to be measured corresponds to one part object contour and one target part contour, where the target part contour of the part to be measured is calculated based on the part object contour of the part to be measured and is the final contour of the part to be measured. It can be understood that the part object contour is a rough contour output by the target recognition model, and the target part contour is the precise contour of the part to be measured.
[0036] S5. Obtain the diagnostic rule matching the target part category where the part to be measured is located, and based on the diagnostic rule and the target part contour of the part to be measured, obtain the fault diagnosis result of the target to be measured.
[0037] In this embodiment, for one target to be measured, it corresponds to one or more part categories, and there is one or more parts under each part category. The diagnostic rules corresponding to each part category are pre-configured, and different part categories can correspond to the same or different diagnostic rules. Target A has two parts to be measured, a1 and a2, and a1 and a2 correspond to two different part categories, A1 and A2 respectively. The diagnostic rule of A is that the highest temperature in the detected part area is higher than 130 degrees, which belongs to a critical fault; the diagnostic rule of A2 is that the lowest temperature in the detected part area is less than 20 degrees, then it belongs to a serious fault.
[0038] The ledger and rule data can be pre-configured. The ledger indicates the user-defined inspection task package, and the data in the ledger includes but is not limited to: the inspection order of the target to be measured, the identifier of the target to be measured. The rule data stores the part categories in each target to be measured, the diagnostic rules corresponding to the part categories in each target to be measured, the identifier of the target to be measured, the identifier of the part category, etc. The rule data can be stored in the form of a rule table. When starting the inspection task, according to the task progress, obtain the relevant information of the target to be measured from the rule data, and this relevant information is used to associate the rule data generated in the first stage.
[0039] In this embodiment, the diagnostic rules corresponding to the same part category in different targets to be measured can be the same or different. The user can formulate the test rules of G_List on the user interface according to actual needs, and record and generate a diagnostic rule table. For example, the part category set of the target to be measured G is TP G =[TP1, TP2], where TP1 represents the first type of part category and TP2 represents the second part category. Then the diagnostic rule of G is [RG_TP1, RG_TP2], where RG_TP1 represents the diagnostic rule corresponding to TP1 in target G, and RG_TP2 represents the diagnostic rule corresponding to TP2 in target G; the part category set of target H is TP H= [TP2, TP3], where TP3 represents the third part category, then the rule for H is [RH_TP2, RH_TP3], RH_TP2 represents the diagnostic rule corresponding to TP2 in target H, and RH_TP3 represents the diagnostic rule corresponding to TP3 in target H; although both target G and H contain the TP2 part category, in different targets, the diagnostic rules for this part category are not related, that is, they can be the same or different. More specifically, for example, target G has two parts to be measured, a and b. a corresponds to part category A1, and b corresponds to part category A2. Among them, in target G, the rule corresponding to A2 is that the highest temperature in the detected part area being higher than 85° belongs to a serious fault. Target H has two parts to be measured, c and d. c corresponds to part category A2, and d corresponds to part category A3; among them, in target H, the rule for A2 is that the highest temperature in the detected part being higher than 55° belongs to a critical fault.
[0040] In this embodiment, for each target part contour, there is a corresponding part to be measured. For each part to be measured, based on the diagnostic rule corresponding to the target part category where the part to be measured is located and the corresponding target part contour, the fault diagnosis result of each part to be measured can be obtained, that is, the fault diagnosis result corresponding to each target part contour can be obtained. According to the fault diagnosis results corresponding to each target part contour, the fault diagnosis result of the target to be measured can be obtained. If the fault diagnosis result corresponding to the part to be measured indicates a fault, it means that the target to be measured has a fault. When the fault diagnosis results corresponding to all target parts to be measured are normal, it means that the fault diagnosis result of the target to be measured is normal. As Figure 3 shown, Figure 3 is a schematic diagram showing the image fault diagnosis result in an embodiment, Figure 3 the parts to be measured of the target to be measured are S00, S01, S02, S03, and it is determined that the diagnostic result of the target to be measured is normal.
[0041] In the above embodiments, the part categories of the target to be measured and the diagnostic rules corresponding to the part categories are pre-configured. By obtaining the infrared image data of the target to be measured, the part object category and the part object contour of the part object of the target to be measured are identified from the infrared image data, and the part to be measured that needs attention is detected, which reduces the difficulty of the algorithm in identifying the target. Based on the configured part category and the identified part object category, the target part category to be finally detected is determined. Based on the target part category, the target part contour is determined. Based on the target part contour, the temperature data of the corresponding part to be measured is determined. Based on the part category corresponding to the target part contour, the diagnostic rule corresponding to the corresponding part to be measured is determined, and the fault diagnosis results corresponding to each target part contour are obtained. According to the fault diagnosis results corresponding to each target part contour, the fault diagnosis result of the target to be measured can be obtained. By associating with the account book, the fault diagnosis result of the target to be measured is determined according to the fault diagnosis results of each part during diagnosis, which increases the flexibility of target recognition; by integrating the diagnostic rules of the target, the knowledge reserve requirements for the users are reduced, the steps of the staff are simplified, the detection efficiency is improved, and convenience is provided for the users.
[0042] In some embodiments, the determining the target part category based on the part category and the part object category includes:
[0043] Taking the intersection of the part category and the part object category as the target part category.
[0044] In this embodiment, the part category is the part category corresponding to the target to be measured pre-configured by the user, and the part object category is the part category identified based on the infrared image data. Some of the part categories that may be identified are not in the part categories that need to be detected. Therefore, it is necessary to determine the part category that finally needs to be detected, that is, the target part category, based on the part category and the part object category, and take the intersection of the part category and the part object category as the target part category. For example, the part category includes category A1 and A2, and the part object category includes A1, A2, and A3, then the target part category is A1 and A2.
[0045] In the above embodiments, for the target to be measured, taking the intersection of the configured part category and the part category identified based on the infrared image data can further determine the part to be measured that needs to be detected in the target to be detected, so as to detect the part that needs attention, improve the detection efficiency, and provide convenience for the users.
[0046] In some embodiments, the determining the target part contour of the part to be measured based on the target part category and the part object contour includes at least one of the following:
[0047] Perform an image processing operation on the contour of the part object to obtain a part mask corresponding to each part object contour. From the part masks corresponding to each part object contour, screen out the part mask corresponding to the part object category that is the same as the target part category, and use the edge of the screened part mask as the target part contour of the part to be measured;
[0048] From the part object contours, screen out the part object contour corresponding to the part object category that is the same as the target part category as the part object contour of the part to be measured. Perform an image processing operation on the part object contour of the part to be measured to obtain a mask corresponding to the part to be measured, and use the edge of the mask corresponding to the part to be measured as the target part contour.
[0049] Optionally, performing an image processing operation on the part object contour to obtain a part mask corresponding to each part object contour includes: calculating the mask corresponding to the part object contour; after processing the mask corresponding to the part object contour by the non-maximum suppression algorithm, and then performing at least one of dilation, erosion processing, and size adjustment to obtain the final mask corresponding to the part object contour as the part mask corresponding to the part object contour. Optionally, the method for obtaining the mask corresponding to the part object contour of the part to be measured by performing an image processing operation on the part object contour of the part to be measured is the same as the method for obtaining the part mask corresponding to the part object contour described above, that is, the screening operation can be performed first and then the image processing operation, or the image processing operation can be performed first and then the screening operation.
[0050] A mask is a binary (or grayscale) image with the same size as the original image and is used to mark the region of interest. The mask corresponding to the part object contour is an image with the same size as the infrared image data. The position in the mask indicating the part object is marked with a first identifier, for example, marked as "1", and the position indicating the non-part object is marked with a second identifier, marked as "0". The non-maximum suppression algorithm (Non-Maximum Suppression, nms) processing refers to removing overlapping boundaries and retaining the most credible one.
[0051] In the above embodiments, the part to be measured is determined from the recognized part objects according to the target part category, and an image processing operation is performed on the part object contour corresponding to the part to be measured to obtain the target part contour of the part to be measured. Using the target part contour to more accurately express the edge contour of the part to be measured can improve the fault diagnosis accuracy of the part to be measured subsequently.
[0052] In some embodiments, the diagnostic rule matching the target part category where the part to be measured is located, based on the diagnostic rule and the target part contour of the part to be measured, obtaining the fault diagnosis result of the target to be measured includes:
[0053] For any part to be measured, extract the temperature data within the target part contour of the part to be measured to obtain the target temperature data of the part to be measured, and calculate the diagnostic temperature of the part to be measured based on the target temperature data.
[0054] According to the diagnostic rule corresponding to the target part category where the part to be measured is located and the diagnostic temperature, obtain the fault diagnosis result of the target to be measured.
[0055] In this embodiment, the diagnostic temperature can be used to calculate and measure whether there is a preset fault in the corresponding target to be measured according to the diagnostic rules corresponding to different targets to be measured. The diagnostic temperature includes at least one of the following: the maximum temperature, the minimum temperature, the average temperature, and the temperature difference within the target part contour. The temperature difference can be a combination of any one or more of the following: the difference between the maximum temperature and the reference temperature, the difference between the minimum temperature and the reference temperature, and the difference between the average temperature and the reference temperature. One target part contour corresponds to one part to be measured, and the target part category where the target part contour is located is the corresponding target part category where the part to be measured is located, that is, compare the diagnostic temperature of the part to be measured with the diagnostic rule corresponding to the target part category where the part to be measured is located to determine the fault diagnosis result of the part to be measured.
[0056] Optionally, the obtaining the fault diagnosis result of the target to be measured according to the diagnostic rule corresponding to the target part category where the part to be measured is located and the diagnostic temperature includes:
[0057] For the target to be measured, when the fault diagnosis result of one part to be measured in the target to be measured indicates a fault, determine that the target to be measured has a fault;
[0058] When the fault diagnosis results of all parts to be measured in the target to be measured indicate normal, determine that the target to be measured is normal.
[0059] In this embodiment, the parts to be measured in the target to be measured can be automatically diagnosed, and an automatic warning is issued when a fault is determined.
[0060] In the above embodiment, extract the temperature data within the target contour of the part to be measured, obtain the diagnostic temperature of the part to be measured, and determine the fault diagnosis result of the part to be measured based on the diagnostic rule corresponding to the target part category where the part to be measured is located and the diagnostic temperature, so as to focus on the parts that need to be concerned in the target to be measured without having to perform all detections, thereby improving the detection efficiency.
[0061] In some embodiments, the target recognition of the infrared image data to determine the part object category and part object contour of the part object in the infrared image data includes:
[0062] Based on a pre-trained object recognition model, perform object recognition on the infrared image data to determine the part object category and part object contour in the infrared image data, where the object recognition model is trained based on a training data set, and each training sample in the training data set includes a part sample image and the part category corresponding to the part sample image.
[0063] In this embodiment, the object recognition model can be a model based on one of the following: RCNN, Fast RCNN, Faster RCNN, YOLO, SSD. The basic principle of RCNN is to use selective search to generate candidate regions, extract features through CNN, and finally classify with SVM; the basic principle of Fast RCNN is to integrate classification and bounding box regression into the same network to improve speed; the basic principle of Faster RCNN is to introduce a Region Proposal Network (RPN) to improve detection efficiency; YOLO can transform object detection into a regression problem, directly predict bounding boxes and class probabilities, is fast, and is suitable for real-time scenarios; SSD can perform detection on feature maps of multiple scales and is suitable for the detection of multi-scale targets. RCNN, Fast RCNN, and Faster RCNN all belong to two-stage object detection algorithms and can be applied to scenarios with higher accuracy requirements, while YOLO and SSD both belong to single-stage object detection algorithms and can be applied to scenarios with higher real-time requirements.
[0064] In this embodiment, the training samples can be obtained by disassembling the images collected for the training target. The training target includes the target to be measured or other targets of the same category as the target to be measured. Each training target is disassembled into one or more parts, and all the obtained parts are classified according to their shapes to obtain a part category set TP A , A is the total number of part categories, and the part category set TP A forms a training data set, and each part in the part category set TP A corresponds to a part sample image and the part category corresponding to the part.
[0065] In the above embodiment, a training data set is formed with each part of the target and the part category corresponding to the part, and the object recognition model is trained. In this way, the trained object recognition model can recognize the local part objects in the target to be measured and can determine the part object category of the recognized part objects, so as to facilitate subsequent focus only on the part to be measured that needs to be detected, thereby improving the detection efficiency.
[0066] In some embodiments, the method further includes:
[0067] According to the coordinate data of the target part contour of the part to be measured, draw and display the target part contour of the part to be measured on the infrared image, and / or
[0068] Display the part-related data of the part to be measured, where the part-related data includes one of the following: part identifier, maximum temperature, minimum temperature, and average temperature within the contour of the target part.
[0069] As Figure 3 shown, the parts to be measured S00, S01, S02, and S03 are part identifiers. Draw and display the contour of the target part of the part to be measured on the infrared image, display the maximum temperature of each part to be measured in the area near the contour of the target part corresponding to each part to be measured, and display the fault diagnosis result of the target to be measured on the infrared image.
[0070] In the above embodiment, displaying the contour of the target part of the part to be measured and / or the part-related data on the infrared image can visually display the information of the part to be measured.
[0071] In some embodiments, the method further includes at least one of the following:
[0072] Based on the part category option provided by the user interface, obtain a category selection instruction for the part category option, and display the fault diagnosis data of the part corresponding to the category selection instruction on the user interface;
[0073] Based on the part category option and part option provided by the user interface, obtain a category selection instruction for the part category option and obtain a part selection instruction for the part option, and display the fault diagnosis data of the part corresponding to the part selection instruction under the category corresponding to the category selection instruction on the user interface.
[0074] In this embodiment, for a target to be measured, the user can select the part category to be viewed through the part category option, and then display the fault diagnosis data of all parts to be measured under the part category on the user interface. Optionally, the user interface can also provide a target to be measured option, and the target to be viewed can be selected through the target to be measured option. The user interface can also provide a part option, and the part to be viewed can be selected through the part option. Through the part category option and part option, the part category to be viewed and the part to be viewed can be selected. For example, the part categories are A1 and A2, and there are three parts S01, S02, and S03 under A1. The A1 can be selected through the part category option, and then S02 can be selected through the part option, so that the fault diagnosis result of the S02 part under A1 can be viewed.
[0075] In the above embodiment, by providing the part category option and part option through the user interface, the user can select these options to select the part to be viewed by himself, so as to view the fault diagnosis result of the local part of the target to be measured.
[0076] In some embodiments, the present application further provides an infrared image fault diagnosis system, which includes an infrared imaging device and a terminal device. The system further includes: The terminal device is used to configure the part categories corresponding to the target to be measured and the diagnosis rules corresponding to each of the part categories, and the infrared imaging device is used to implement the infrared image fault diagnosis method provided in any embodiment of the present application.
[0077] In this embodiment, the terminal device is used to implement the configuration operation in the first stage, that is, to configure the part categories corresponding to the target to be measured and the diagnosis rules corresponding to each part category. The terminal device includes, but is not limited to: devices such as computer devices, mobile phones, tablets, and various wearable devices.
[0078] In some embodiments, the terminal device is used to obtain a target infrared image including the target to be measured, perform preliminary disassembly and classification on the target infrared image to obtain preliminary disassembly part data, and display the preliminary disassembly part data on the terminal user interface, where the preliminary disassembly part data includes the initial contours of each part and the initial part categories corresponding to each part;
[0079] Based on the terminal user interface, obtain the part to be measured configured, obtain the adjustment data for the initial contour of the part to be measured based on the terminal user interface to obtain the final contour corresponding to the part to be measured, and obtain the adjustment data for the initial part category of the part to be measured to obtain the part category corresponding to the part to be measured.
[0080] In this embodiment, image processing algorithms can be used to identify the target in the target infrared image, and perform preliminary disassembly on the target to obtain the initial contours of the various parts obtained by preliminary disassembly, and classify the various parts according to the shapes of the various parts to obtain the initial part categories. Since there may be some errors in automatic disassembly, in order to improve the accuracy of subsequent recognition, contour adjustment controls can be provided through the terminal user interface. The user adjusts the initial contour through the contour adjustment control, and obtains the adjustment data for the initial contour through the terminal user interface during the process of adjusting the contour to obtain the final contour corresponding to the part to be measured. The terminal user interface can also provide part category adjustment controls. The user adjusts the initial part category of the part to be measured through the part category adjustment control, and obtains the adjustment data for the initial part category of the part to be measured during the process of adjusting the initial part category to obtain the part category corresponding to the part to be measured. The control forms provided on the terminal user interface include, but are not limited to: button icons, radio boxes, drop-down boxes, input boxes, etc. Optionally, this step can be applied to the acquisition process of the training dataset.
[0081] In the above embodiments, the parts of each target in the target infrared image can be initially disassembled automatically, and then the results of the initial disassembly can be fine-tuned through the user interface, so as to improve the accuracy of subsequent recognition.
[0082] In some embodiments, when the terminal device obtains a trigger instruction for configuring the diagnostic rules of the target to be measured, it enters the display interface of the diagnostic rules of the target to be measured, and supports configuration operations such as modifying, adding, and deleting at least one of the diagnostic rules corresponding to each part category of the target to be measured.
[0083] In this embodiment, through the setting of the viewing icon of the diagnostic rules, the user can click on the viewing icon of the diagnostic rules to enter the display interface of the diagnostic rules of the target to be measured, and support the user to perform configuration operations such as modifying, adding, and deleting at least one of the diagnostic rules corresponding to each part category.
[0084] In the above embodiments, the diagnostic rules corresponding to each part category of the target to be measured can be configured through the user interface, so as to improve the intelligence of the device and the user experience.
[0085] In some embodiments, the following is an application example of an infrared image fault diagnosis method, including the following steps:
[0086] Step0: Start the inspection task, and obtain relevant information of the target G to be measured according to the task progress, and this information is used to associate with a pre-generated diagnostic rule table.
[0087] Step1: Obtain the temperature data set T mn and the image data set Y mn .
[0088] Step2: Send the image data set Y mn to the target recognition model to identify the part object category and part object contour.
[0089] Identify all classified part object contour sets PART X and part object category sets TP Y in the image, determine the mask mask and bounding box bbox of each part object contour image PART X1 , where X1 ∈ X. Dilate (cv.dilate) or erode (cv.erode) the bbox and mask, and adjust the size and shape of the PART X1 area to obtain the final mask Mask_final_PART X1, X1∈X, where cv2.dilate and cv2.erode are functions in OPENCV. All masks are obtained as above, and the part mask set Mask_final_PART is obtained. X , each part PART has a mask information and part category information TP.
[0090] Step 3: According to the information of the target G to be tested, obtain the part category set TP of G G ;
[0091] Calculate TP G ∩TP Y =TP Y_G , from the part object contour set PART X and its part mask set Mask_final_PART X Filter out categories in TP Y_G Part object outline set in the collection PART TP_Y_G and its part mask set and Mask_final_PART TP_Y_G .
[0092] Step 4: Mask_final_PART based on the selected part to be tested TP_Y_G1 Use the findContours function in OPENCV to calculate the target part contour PART TP_Y_G1 The coordinate set of the edge Area TP_Y_G1 [int], and move the edge at Y mn Draw it out on the surface and mark each part to be tested (such as S01, S02, S03...) as shown in the attached figure. Figure 3 Middle edge contour. Mask_final_PART TP_Y_G1 ∈Mask_final_PART TP_Y_G , as above, draw the part set PART TP_Y_G all edges.
[0093] Step 5. In T mn According to the target part contour PART TP_Y_G1 Mask_final_PART TP_Y_G1 Extract all the temperatures of the measured part to obtain the temperature set T TP_Y_G1 , and calculate the temperature data required for diagnosis, such as the maximum temperature T within the contour of the target part TP_Y_G1 .max, the minimum temperature T within the target area contour TP_Y_G1 .min, the average temperature T within the contour of the target area TP_Y_G1 .avg,ΔT TP_Y_G1As described above, the part object contour set PART is calculated. X The corresponding temperature set T TP_Y_G and the temperature data required for diagnosis. So far, each part to be measured PAPT has a set of temperature data and a part category information TP.
[0094] Step6. Obtain the part category TP from the diagnostic rule table in the information of the target G in Step0. G The corresponding fault rule set RG.
[0095] Step7. Calculate the temperature data t (t ∈ T TP_Y_G ) of each part to be measured, the part category tp (tp ∈ TP G ) and the diagnostic rule RG information to obtain the fault diagnosis result of each part to be measured.
[0096] In some embodiments, the present application is divided into the following steps: a configuration step, a data acquisition step, a data storage step, a target classification and disassembly step, an intelligent recognition step, a screening and display step, and an intelligent diagnosis step. 1) Configuration operation: Configure the part categories corresponding to the target to be measured and the diagnostic rules corresponding to each of the part categories; 2) Data acquisition step: used to acquire image data, including infrared temperature data and infrared image data. 3) Data storage step: used to store all data such as images and temperatures, as well as the data that needs to be temporarily stored during the data processing of the cache. 4) Intelligent recognition step: The device processes the acquired infrared image data through an image recognition algorithm to identify the part object category and the part object contour. 5) Screening and display step: According to the information of the target to be measured, and in accordance with the records of the configuration step, screen out the set of parts to be measured of the target to be measured, and draw the contours of these sets of parts to be measured for temperature measurement and diagnosis. 6) Intelligent diagnosis step: Search for the corresponding diagnostic rules according to the information of the target to be measured; calculate the temperature values corresponding to the rules through the target part contours of the parts to be measured identified by modules 4) and 5), and automatically perform defect diagnosis according to the part type rule characteristics and temperature values, and automatically give the diagnosis result for early warning.
[0097] At least one embodiment of the present application proposes an intelligent fault diagnosis solution based on an infrared thermal imager. Relying on the infrared thermal imager, an intelligent fault diagnosis module is added. By disassembling and classifying the parts of the target to be measured, the workload of algorithm model training is reduced, and at the same time, the difficulty of the algorithm for identifying the target is reduced; by associating with the account book, the fault diagnosis result of the target to be measured is determined according to the fault diagnosis results of each part during diagnosis, increasing the flexibility of target recognition; by carefully drawing the contour of the target to be measured, the heat interference of non-targets is reduced. By integrating the diagnostic rules of the target, the requirement for the knowledge reserve of the user is reduced, the steps of the staff are simplified, the detection efficiency is improved, and convenience is provided for the user.
[0098] On the other hand, the present application provides a computer program product, including a computer program which, when executed by a processor, implements the infrared image fault diagnosis method according to any embodiment of the present application.
[0099] Among them, in the computer program product, an optional implementation form of the program module architecture of the computer program for implementing each step of the infrared image fault diagnosis method may be an infrared image fault diagnosis device.
[0100] The above mainly introduces the solution provided by the embodiments of the present application from the perspective of the method. To implement the above functions, the infrared image fault diagnosis device includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraint conditions of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.
[0101] The embodiments of the present application can, according to the above method, exemplarily divide the functional modules of the infrared image fault diagnosis device. For example, the infrared image fault diagnosis device may include each functional module corresponding to each functional division, or two or more functions may be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there may be other division methods in actual implementation.
[0102] Please refer to Figure 4 , an embodiment of the present application provides an infrared image fault diagnosis device, including: an acquisition module 41, configured to acquire infrared image data collected for a target to be measured; the acquisition module 41 is further configured to acquire the part category corresponding to the target to be measured; an identification module 42, configured to perform target identification on the infrared image data to determine the part object category and part object contour of the part object in the infrared image data; a detection module 43, configured to determine the target part category based on the part category and the part object category, and determine the target part contour of the part to be measured based on the target part category and the part object contour; the detection module 43 is further configured to acquire a diagnosis rule matching the target part category where the part to be measured is located, and obtain a fault diagnosis result of the target to be measured based on the diagnosis rule and the target part contour of the part to be measured.
[0103] Optionally, the detection module 43 is further configured to:
[0104] Use the intersection of the part category and the part object category as the target part category.
[0105] Optionally, the detection module 43 is further configured to:
[0106] Perform an image processing operation on the part object contour to obtain a part mask corresponding to each part object contour. From the part masks corresponding to each part object contour, screen out the part masks corresponding to the part object categories that are the same as the target part category, and use the edge of the screened part mask as the target part contour of the part to be measured;
[0107] From the part object contours, screen out the part object contours corresponding to the part object categories that are the same as the target part category as the part object contours of the part to be measured. Perform an image processing operation on the part object contours of the part to be measured to obtain a mask corresponding to the part to be measured, and use the edge of the mask corresponding to the part to be measured as the target part contour.
[0108] Optionally, the detection module 43 is further configured to:
[0109] For any part to be measured, extract the temperature data within the target part contour of the part to be measured to obtain the target temperature data of the part to be measured, and calculate the diagnostic temperature of the part to be measured based on the target temperature data;
[0110] According to the diagnostic rule corresponding to the target part category where the part to be measured is located and the diagnostic temperature, obtain the fault diagnosis result of the target to be measured.
[0111] Optionally, the detection module 43 is further configured to:
[0112] For the target to be measured, when the fault diagnosis result of one part to be measured in the target to be measured indicates a fault, determine that the target to be measured has a fault;
[0113] When the fault diagnosis results of all parts to be measured in the target to be measured indicate normal, determine that the target to be measured is normal.
[0114] Optionally, the recognition module 42 is further configured to:
[0115] Based on a pre-trained target recognition model, perform target recognition on the infrared image data to determine the part object category and part object contour in the infrared image data, where the target recognition model is trained based on a training data set, and each training sample in the training data set includes a part sample image and the part category corresponding to the part sample image.
[0116] Optionally, it further includes a display module 44, configured to:
[0117] According to the coordinate data of the target part contour of the part to be measured, draw and display the target part contour of the part to be measured on the infrared image, and / or
[0118] Display the part-related data of the part to be measured, where the part-related data includes one of the following: part identifier, maximum temperature, minimum temperature, average temperature, temperature difference within the target part contour.
[0119] Optionally, the display module 44 is further configured to:
[0120] Based on the part category option provided by the user interface, obtain a category selection instruction for the part category option, and display the fault diagnosis data of the part corresponding to the category selection instruction on the user interface;
[0121] Based on the part category option and part option provided by the user interface, obtain a category selection instruction for the part category option and obtain a part selection instruction for the part option, and display the fault diagnosis data of the part corresponding to the part selection instruction under the category corresponding to the category selection instruction on the user interface.
[0122] Please refer to Figure 5 , on the other hand, an infrared imaging device 10 according to an embodiment of the present application further includes an infrared image acquisition device 12, a processor 13, and a memory 14. The memory 14 stores a computer program. When the computer program is executed by the processor, the processor 13 executes the steps of an infrared image fault diagnosis method provided in any one of the above embodiments of the present application.
[0123] The processor 13 is a control center, which connects various parts of the entire infrared imaging device through various interfaces and lines, and executes various functions of the infrared imaging device and processes data by running or executing software programs and / or modules stored in the memory 14, and calling data stored in the memory 14. Optionally, the processor 13 may include one or more processing cores; preferably, the processor 13 may integrate an application processor and a modulation and demodulation processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, etc., and the modulation and demodulation processor mainly processes wireless communication. It can be understood that the above modulation and demodulation processor may not be integrated into the processor 13.
[0124] The memory 14 can be used to store software programs and modules. By running the software programs and modules stored in the memory 14, the processor 13 can execute various functional applications and data processing. The memory 14 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of an infrared imaging device. In addition, the memory 14 can include high-speed random access memory and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 14 can also include a memory processor to provide the processor 13 with access to the memory 14.
[0125] In another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is caused to execute the steps of an infrared image fault diagnosis method provided in any one of the above embodiments of the present application.
[0126] In another aspect of the embodiments of the present application, there is provided a computer program product including a computer program, where the computer program, when executed by a processor, implements an infrared image fault diagnosis method as described in any one of the embodiments of the present application.
[0127] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods provided in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or an external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.
[0128] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. The protection scope of the present invention shall be subject to the protection scope of the claims described.
Claims
1. An infrared image fault diagnosis method, characterized in that, Including: Obtaining infrared image data collected for a target to be measured; Obtaining the part category corresponding to the target to be measured; Performing target recognition on the infrared image data to determine the part object category and part object contour of the part object in the infrared image data; Determining the target part category based on the part category and the part object category, and determining the target part contour of the part to be measured based on the target part category and the part object contour; Obtaining a diagnostic rule matching the target part category where the part to be measured is located, and obtaining a fault diagnosis result of the target to be measured based on the diagnostic rule and the target part contour of the part to be measured.
2. The infrared image fault diagnosis method according to claim 1, wherein The determining the target part category based on the part category and the part object category includes: Taking the intersection of the part category and the part object category as the target part category.
3. The infrared image fault diagnosis method according to claim 1, wherein The determining the target part contour of the part to be measured based on the target part category and the part object contour includes at least one of the following: Performing an image processing operation on the part object contour to obtain a part mask corresponding to each part object contour, screening out the part mask corresponding to the part object category that is the same as the target part category from the part masks corresponding to each part object contour, and taking the edge of the screened part mask as the target part contour of the part to be measured; Screening out the part object contour corresponding to the part object category that is the same as the target part category from the part object contours as the part object contour of the part to be measured, performing an image processing operation on the part object contour of the part to be measured to obtain a mask corresponding to the part to be measured, and taking the edge of the mask corresponding to the part to be measured as the target part contour.
4. The infrared image fault diagnosis method according to claim 1, characterized in that The diagnostic rule matching the target part category where the part to be measured is located, and obtaining a fault diagnosis result of the target to be measured based on the diagnostic rule and the target part contour of the part to be measured includes: For any part to be measured, extracting the temperature data within the target part contour of the part to be measured to obtain the target temperature data of the part to be measured, and calculating the diagnostic temperature of the part to be measured based on the target temperature data; Obtaining a fault diagnosis result of the target to be measured according to the diagnostic rule corresponding to the target part category where the part to be measured is located and the diagnostic temperature.
5. The infrared image fault diagnosis method according to claim 4, characterized in that, The obtaining a fault diagnosis result of the target to be measured according to the diagnostic rule corresponding to the target part category where the part to be measured is located and the diagnostic temperature includes: For the target to be measured, when the fault diagnosis result of one part to be measured in the target to be measured indicates a fault, determining that the target to be measured has a fault; When the fault diagnosis results of all parts to be measured in the target to be measured indicate normal, determining that the target to be measured is normal.
6. The infrared image fault diagnosis method according to claim 1, wherein The performing target recognition on the infrared image data to determine the part object category and part object contour of the part object in the infrared image data includes: Based on a pre-trained object recognition model, perform object recognition on the infrared image data to determine the part object category and part object contour in the infrared image data, where the object recognition model is trained based on a training data set, and each training sample in the training data set includes a part sample image and the part category corresponding to the part sample image.
7. The infrared image fault diagnosis method according to claim 1, wherein, The method further includes: According to the coordinate data of the target part contour of the part to be measured, draw and display the target part contour of the part to be measured on the infrared image, and / or Display the part association data of the part to be measured, where the part association data includes one of the following: part identifier, maximum temperature, minimum temperature, average temperature, temperature difference within the target part contour.
8. The infrared image fault diagnosis method according to claim 1, wherein, The method further includes at least one of the following: Based on the part category options provided by the user interface, obtain a category selection instruction for the part category options, and display the fault diagnosis data of the part corresponding to the category selection instruction on the user interface; Based on the part category options and part options provided by the user interface, obtain a category selection instruction for the part category options and obtain a part selection instruction for the part options, and display the fault diagnosis data of the part corresponding to the part selection instruction under the category corresponding to the category selection instruction on the user interface.
9. An infrared imaging device, characterized in that, Includes a memory, a processor, and an infrared image acquisition device; The infrared image acquisition device acquires infrared image data; The processor stores a computer program therein, and the processor is configured to execute the computer program to implement the infrared image fault diagnosis method according to any one of claims 1 to 8.
10. An infrared image fault diagnosis system, characterized in that, Includes the infrared imaging device and the terminal device as described in claim 9, and the terminal device is configured to configure the part category corresponding to the target to be measured and the diagnosis rules corresponding to each of the part categories.
11. The infrared image fault diagnosis system according to claim 10, wherein, The terminal device acquires a target infrared image including the target to be measured, performs preliminary disassembly and classification on the target infrared image to obtain preliminary disassembly part data, and displays the preliminary disassembly part data on the terminal user interface, where the preliminary disassembly part data includes the initial contour of each part and the initial part category corresponding to each part; Based on the part to be measured configured through the terminal user interface, obtain adjustment data for the initial contour of the part to be measured based on the infrared image data to obtain the final contour corresponding to the part to be measured, and obtain adjustment data for the initial part category of the part to be measured to obtain the part category corresponding to the part to be measured.
12. The infrared image fault diagnosis system according to claim 11, wherein, When the terminal device obtains a trigger instruction for configuring the diagnosis rules of the target to be measured, it enters the display interface of the diagnosis rules of the target to be measured, and supports configuration operations such as modifying, adding, and deleting at least one of the diagnosis rules corresponding to each part category of the target to be measured.
13. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by the processor, implements an infrared image fault diagnosis method according to any one of claims 1 to 8.
14. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, it implements an infrared image fault diagnosis method according to any one of claims 1 to 9.