Infrared image segmentation method and system based on visible light auxiliary object recognition

Abstract

The invention discloses an infrared image segmentation method and system based on visible light auxiliary object recognition, and belongs to the technical field of equipment fault detection, and the method comprises the steps: carrying out the fault feature recognition of a visible light image based on an infrared image and the visible light image of equipment, obtaining a bounding box and a class label for recognizing the fault features, and generating a recognition result; and performing image registration on the infrared image and the visible light image, mapping an identification result into the infrared image, and performing image segmentation through a deep learning model with a multi-scale feature extraction module. According to the method, the object recognition result of the visible light image and the fault features of the infrared image are combined, the precision and adaptability of infrared image segmentation are improved, the method can be effectively applied to fault detection of a power system, and the method has wide application prospects and high innovativeness.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment fault detection, and in particular to an infrared image segmentation method and system based on visible light-assisted object recognition. Background Art

[0002] With the application of infrared imaging technology in power systems, the number and complexity of infrared images are increasing. Traditional manual analysis methods are inefficient and prone to errors. Most existing automatic detection methods rely on simple image segmentation technology, which has poor adaptability and cannot fully utilize the regional characteristics of infrared images. In addition, existing infrared image segmentation methods lack accuracy and adaptability in complex environments. It is difficult to accurately distinguish between fault areas and other high-temperature areas by simply relying on infrared images for segmentation. It is urgent to design an infrared image segmentation technology that combines visible light images with object recognition to improve segmentation accuracy and adaptability. Summary of the invention

[0003] In order to solve the above problems, the purpose of the present invention is to provide an infrared image segmentation technology combined with visible light image assisted object recognition, aiming to improve the accuracy and adaptability of infrared image segmentation by combining the object recognition results of visible light images and the fault characteristics of infrared images, and to be able to be effectively applied to fault detection in power systems, so that the technology has broad application prospects and strong innovation.

[0004] In order to achieve the above technical objectives, the present application provides an infrared image segmentation method based on visible light assisted object recognition, comprising the following steps:

[0005] Based on the infrared image and visible light image of the device, perform object recognition on the visible light image, obtain the bounding box and category label of the recognized object, and generate a recognition result, wherein the object represents the fault feature;

[0006] The infrared image and the visible light image are registered, the recognition result is mapped to the infrared image, and the image is segmented using a deep learning model with a multi-scale feature extraction module.

[0007] Preferably, in the process of acquiring the infrared image and the visible light image of the device, the acquired initial infrared image and the initial visible light image of the device are subjected to denoising, normalization and enhancement processing to generate the infrared image and the visible light image of the device.

[0008] Preferably, in the process of object recognition, a pre-trained object detection model is used to perform object recognition on the visible light image to identify the devices and other important objects in the image; the position of the identified object is marked, and the bounding box and category label of the object are generated, wherein the pre-trained object detection model includes one of YOLO, Faster or R-CNN.

[0009] Preferably, in the process of performing image registration on the infrared image and the visible light image, image registration is performed by using a feature point matching technology SIFT, wherein SIFT is expressed as:

[0010]

[0011] Among them, f V (p i ) and f IR (q i ) represent the feature points in the visible light image and infrared image respectively, and N is the number of feature points.

[0012] Preferably, in the process of mapping the recognition result into the infrared image, the position of the object in the visible light image is (x V ,y V ), the position mapped to the infrared image is (x IR ,y IR ):

[0013] (x IR ,y IR )=H·(x V ,y V );

[0014] Among them, H is the transformation matrix for image registration.

[0015] Preferably, in the process of performing image segmentation on the infrared image, a deep learning model is used to segment the infrared image, and the model input is the infrared image I IR And the fusion feature F of the object recognition result:

[0016] F = Concat(I IR , R V→IR );

[0017] Among them, RV→IR is the object recognition result after mapping, and Concat represents the feature concatenation operation;

[0018] The segmentation model extracts features through convolution operations and generates segmentation results S:

[0019] S=SegNet(F).

[0020] Preferably, in the process of segmenting the infrared image by the deep learning model, a multi-task loss function is used to optimize the segmentation model, including a pixel-level cross entropy loss L CE and boundary loss L B :

[0021] L=L CE +λL B;

[0022] Among them, L CE is the cross entropy loss, L B is the boundary loss, and λ is the weight parameter.

[0023] Preferably, after the infrared image is segmented by the deep learning model, the segmentation result is post-processed by removing noise and artifacts.

[0024] The present invention discloses an infrared image segmentation system based on visible light assisted object recognition, comprising:

[0025] An image acquisition module, used to acquire infrared images and visible light images of the device;

[0026] An object recognition module is used to perform object recognition on the visible light image based on the infrared image and the visible light image of the device, obtain a bounding box and a category label of the recognized object, and generate a recognition result, wherein the object represents a fault feature;

[0027] The image segmentation module is used to align the infrared image and the visible light image, map the recognition result to the infrared image, and perform image segmentation through a deep learning model with a multi-scale feature extraction module.

[0028] Preferably, the image acquisition module is also used to acquire an initial infrared image and an initial visible light image of the device, and generate an infrared image and a visible light image of the device after denoising, normalization and enhancement processing.

[0029] The present invention discloses the following technical effects:

[0030] (1) High adaptability: Combined with the object recognition results of visible light images, the segmentation method can adaptively identify and segment different devices, improving adaptability.

[0031] (2) High precision: Targeted segmentation reduces the false detection rate and missed detection rate, and improves the accuracy of segmentation, especially in complex backgrounds and changing environments.

[0032] (3) High robustness: The fusion of multimodal data and multi-scale feature extraction technology improve the robustness of the segmentation model, enabling it to cope with complex backgrounds and changing environmental conditions.

[0033] (4) Intelligence: It has a high degree of automation, which reduces the reliance on manual parameter adjustment and is suitable for automatic processing of large-scale infrared images. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0036] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0037] like Figure 1 As shown, the present invention provides an infrared image segmentation method based on visible light assisted object recognition, comprising the following steps:

[0038] Based on the infrared image and visible light image of the device, perform object recognition on the visible light image, obtain the bounding box and category label of the recognized object, and generate a recognition result, wherein the object represents the fault feature;

[0039] The infrared image and the visible light image are registered, the recognition result is mapped to the infrared image, and the image is segmented using a deep learning model with a multi-scale feature extraction module.

[0040] Further preferably, the infrared image segmentation method based on visible light assisted object recognition provided by the present invention, in the process of acquiring the infrared image and visible light image of the device, performs denoising, normalization and enhancement processing on the collected initial infrared image and initial visible light image of the device to generate the infrared image and visible light image of the device.

[0041] Further preferably, in the process of object recognition, the infrared image segmentation method based on visible light assisted object recognition provided by the present invention uses a pre-trained object detection model to perform object recognition on the visible light image, identify the devices and other important objects in the image; mark the positions of the identified objects, and generate the bounding box and category label of the object, wherein the pre-trained object detection model includes one of YOLO, Faster or R-CNN.

[0042] Further preferably, in the infrared image segmentation method based on visible light assisted object recognition provided by the present invention, in the process of image registration of the infrared image and the visible light image, image registration is performed by feature point matching technology SIFT, wherein SIFT is expressed as:

[0043]

[0044] Among them, f V (p i ) and f IR (q i ) represent the feature points in the visible light image and infrared image respectively, and N is the number of feature points.

[0045] Further preferably, in the process of mapping the recognition result to the infrared image, the infrared image segmentation method based on visible light assisted object recognition provided by the present invention converts the position of the object in the visible light image into (x V ,y V ), the position mapped to the infrared image is (x IR ,y IR ):

[0046] (x IB ,y IB )=H·(x V ,y V );

[0047] Among them, H is the transformation matrix for image registration.

[0048] Further preferably, in the infrared image segmentation method based on visible light assisted object recognition provided by the present invention, a deep learning model is used to segment the infrared image during the image segmentation process of the infrared image, and the model input is the infrared image I IR And the fusion feature F of the object recognition result:

[0049] F = Concat(I IR , R V→IR );

[0050] Among them, RV→IR is the object recognition result after mapping, and Concat represents the feature concatenation operation;

[0051] The segmentation model extracts features through convolution operations and generates segmentation results S:

[0052] S=SegNet(F).

[0053] Further preferably, in the infrared image segmentation method based on visible light assisted object recognition provided by the present invention, in the process of segmenting the infrared image through the deep learning model, a multi-task loss function is used to optimize the segmentation model, including a pixel-level cross entropy loss L CE and boundary loss L B :

[0054] L=L CE +λL B ;

[0055] Among them, L CE is the cross entropy loss, L B is the boundary loss, and λ is the weight parameter.

[0056] Further preferably, the infrared image segmentation method based on visible light assisted object recognition provided by the present invention performs post-processing on the segmentation result by removing noise and artifacts after the infrared image is segmented by a deep learning model.

[0057] The present invention discloses an infrared image segmentation system based on visible light assisted object recognition, which is used to implement the infrared image segmentation method based on visible light assisted object recognition provided by the present invention, comprising:

[0058] An image acquisition module, used to acquire infrared images and visible light images of the device;

[0059] An object recognition module is used to perform object recognition on the visible light image based on the infrared image and the visible light image of the device, obtain a bounding box and a category label of the recognized object, and generate a recognition result, wherein the object represents a fault feature;

[0060] The image segmentation module is used to align the infrared image and the visible light image, map the recognition result to the infrared image, and perform image segmentation through a deep learning model with a multi-scale feature extraction module.

[0061] Further preferably, the image acquisition module of the infrared image segmentation system based on visible light assisted object recognition disclosed in the present invention is also used to acquire the initial infrared image and initial visible light image of the device, and generate the infrared image and visible light image of the device after denoising, normalization and enhancement processing.

[0062] Embodiment: The present invention provides an innovative infrared image segmentation method, which makes infrared image segmentation more accurate and efficient by combining the object recognition results of a visible light camera. Specifically, the method includes the following steps:

[0063] 1. Image acquisition and preprocessing: Acquire the infrared image and visible light image of the device at the same time, and perform denoising, normalization and enhancement on the infrared image and visible light image to improve the image quality and provide a good foundation for subsequent processing;

[0064] 2. Object recognition in visible light images: Use pre-trained object detection models (such as YOLO, FasterR-CNN, etc.) to perform object recognition on visible light images to identify devices and other important objects in the image; annotate the locations of the identified objects and generate bounding boxes and category labels for the objects;

[0065] 3. Mapping object recognition results to infrared images: Map the object recognition results in the visible light image to the infrared image. Use image registration technology (such as SIFT, ORB, etc.) to ensure that the visible light image and the infrared image are aligned in space. Mark the object recognition area in the infrared image to provide guidance information for subsequent segmentation;

[0066] 4. Targeted infrared image segmentation: Use the mapped object recognition results to perform targeted segmentation on the infrared image. Focus on detailed segmentation of the marked area, while ignoring irrelevant areas to reduce the false detection rate. Use deep learning models (such as U-Net, SegNet, etc.) to segment the infrared image. The model input is the fusion features of the infrared image and the object recognition result;

[0067] 5. Multi-scale feature fusion: Introduce a multi-scale feature extraction module into the segmentation model to extract image features from different scales and fuse them. Multi-scale feature extraction can capture fault areas of different sizes and shapes, improving the comprehensiveness and accuracy of segmentation;

[0068] 6. Post-processing: Post-process the segmentation results to remove noise and artifacts to further improve the accuracy and robustness of the segmentation.

[0069] Specifically, the detailed technical implementation process of the infrared image segmentation method based on visible light assisted object recognition is as follows:

[0070] (1) Image registration formula:

[0071] Through image registration technology, the visible light image I V With infrared image I IR Align. The registration process uses the feature point matching technology SIFT (Scale Invariant Feature Transform):

[0072]

[0073] Among them, f V (p i ) and f IR (q i ) represent the feature points in the visible light image and infrared image respectively, and N is the number of feature points.

[0074] (2) Object recognition result mapping:

[0075] The object recognition result R in the visible light image V Mapping to infrared image I IR Assume that the position of the object in the visible light image is (x V ,y V ), the position mapped to the infrared image is (x IR ,y IR ):

[0076] (x IR ,y IR )=H·(x V ,y V );

[0077] Among them, H is the transformation matrix for image registration.

[0078] (3) Deep learning model segmentation:

[0079] Use deep learning model to segment infrared images. The model input is infrared image I IR And the fusion feature F of the object recognition result:

[0080] F = Concat(I IR , R V→IR );

[0081] Among them, R V→IR is the object recognition result after mapping, and Concat represents the feature concatenation operation. The segmentation model extracts features through convolution operation and generates the segmentation result S:

[0082] S = SegNet(F);

[0083] (4) Loss function:

[0084] Optimize the segmentation model using a multi-task loss function, including pixel-level cross entropy loss L CE and boundary loss L B :

[0085] L=L CE +λL B ;

[0086] Among them, L CEis the cross entropy loss, L B is the boundary loss, and λ is the weight parameter.

[0087] The present invention fuses the information of visible light images and infrared images through multimodal fusion technology, and uses the object recognition results of visible light images to guide the segmentation of infrared images, thereby improving the accuracy and adaptability of segmentation.

[0088] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0089] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0090] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An infrared image segmentation method based on visible light assisted object recognition, characterized in that: The following steps are involved: Based on the infrared image and the visible light image of the device, perform object recognition on the visible light image, obtain a bounding box and a category label of the recognized object, and generate a recognition result, wherein the object represents a fault feature; The infrared image and the visible light image are registered, the recognition result is mapped into the infrared image, and image segmentation is performed through a deep learning model with a multi-scale feature extraction module.

2. The infrared image segmentation method based on visible light assisted object recognition according to claim 1, characterized in that: In the process of acquiring the infrared image and the visible light image of the device, the acquired initial infrared image and the initial visible light image of the device are subjected to denoising, normalization and enhancement processing to generate the infrared image and the visible light image of the device.

3. The infrared image segmentation method based on visible light assisted object recognition according to claim 2, characterized in that: In the process of object recognition, a pre-trained object detection model is used to perform object recognition on the visible light image to identify the devices and other important objects in the image; the position of the identified object is marked, and the bounding box and category label of the object are generated, wherein the pre-trained object detection model includes one of YOLO, Faster or R-CNN.

4. The infrared image segmentation method based on visible light assisted object recognition according to claim 3, characterized in that: In the process of registering the infrared image and the visible light image, the image registration is performed by using the feature point matching technology SIFT, where SIFT is expressed as: Among them, fV(p i ) and f IR (q i ) represent the feature points in the visible light image and infrared image respectively, and N is the number of feature points.

5. The infrared image segmentation method based on visible light assisted object recognition according to claim 4, characterized in that: In the process of mapping the recognition result into the infrared image, the position of the object in the visible light image is (x V ,y V ), the position mapped to the infrared image is (x IR ,y IR ): (x IR ,y IR )=H·(x V ,y V ); Among them, H is the transformation matrix for image registration.

6. The infrared image segmentation method based on visible light assisted object recognition according to claim 5, characterized in that: In the process of image segmentation of infrared images, a deep learning model is used to segment the infrared images. The model input is the infrared image I IR And the fusion feature F of the object recognition result: F=Concat(I IR ,R V→IR ); Among them, R V→IR is the object recognition result after mapping, Concat represents the feature concatenation operation; The segmentation model extracts features through convolution operations and generates segmentation results S: S=SegNet(F).

7. The infrared image segmentation method based on visible light assisted object recognition according to claim 6, characterized in that: In the process of segmenting infrared images through deep learning models, a multi-task loss function is used to optimize the segmentation model, including the cross entropy loss L at the pixel level. CE and boundary loss L B : L=L CE +λL B ; Among them, L CE is the cross entropy loss, L B is the boundary loss, and λ is the weight parameter.

8. The infrared image segmentation method based on visible light assisted object recognition according to claim 7, characterized in that: After the process of segmenting the infrared image through the deep learning model, the segmentation results are post-processed by removing noise and artifacts.

9. An infrared image segmentation system based on visible light assisted object recognition, characterized in that: include: An image acquisition module, used to acquire infrared images and visible light images of the device; An object recognition module, configured to perform object recognition on the visible light image based on the infrared image and the visible light image of the device, obtain a bounding box and a category label of the recognized object, and generate a recognition result, wherein the object represents a fault feature; An image segmentation module is used to perform image registration on the infrared image and the visible light image, map the recognition result to the infrared image, and perform image segmentation through a deep learning model with a multi-scale feature extraction module.

10. The infrared image segmentation system based on visible light assisted object recognition according to claim 9, characterized in that: The image acquisition module is also used to acquire an initial infrared image and an initial visible light image of the device, and generate an infrared image and a visible light image of the device after performing denoising, normalization and enhancement processing.

Images