An instance segmentation method based on visible and infrared image feature fusion

Abstract

This invention relates to the field of image analysis technology and discloses an instance segmentation method and system based on visible light and infrared image feature fusion. The method includes: acquiring infrared and visible light images in a low-light environment; analyzing the visible light image using a first model to obtain an instance segmentation result in a first state; analyzing the infrared image using a second model to obtain an instance segmentation result in a second state; aligning the instance segmentation results in the first and second states to adjust for heterogeneity between the visible light and infrared images; and generating a final instance segmentation result based on the analysis results. This method significantly improves the accuracy and robustness of instance segmentation in low-light or weak-light environments. It overcomes the shortcomings of traditional methods in low-light environments, effectively reduces alignment errors caused by heterogeneity, improves image quality and detail preservation, and provides reliable technical support for image analysis tasks in practical applications.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, specifically to an instance segmentation method based on visible light and infrared image feature fusion. Background Technology

[0002] With the development of computer vision technology, image segmentation technology is increasingly widely used in fields such as autonomous driving, intelligent monitoring, and medical image processing. However, traditional image segmentation methods typically rely on visible light images, which makes their performance unsatisfactory in low-light or dim-light environments. Visible light images in low-light environments often suffer from insufficient brightness and increased noise, leading to a loss of image details and thus affecting segmentation results. In such cases, traditional image processing methods face significant challenges when processing these images.

[0003] To address this issue, infrared imaging, as an imaging method that can effectively provide information about objects even in low-light conditions, has gradually attracted the attention of researchers. Unlike visible light images, infrared images capture the heat radiated by objects, providing the outline and temperature changes of objects even in complete darkness. Therefore, infrared images exhibit stronger robustness and adaptability in low-light environments. However, infrared images are relatively lacking in detail and lack the color and shape information found in visible light images, making image segmentation using infrared images alone challenging.

[0004] To overcome this challenge, multimodal image fusion technology has been widely used in recent years. By fusing visible light and infrared images, the advantages of both can be combined to improve the accuracy and robustness of image segmentation. Visible light images provide rich information on shape, color, and texture, while infrared images provide crucial information about the thermal radiation of objects. Through methods such as feature-level fusion and decision-level fusion, these two types of information can be combined, thereby improving the accuracy of object detection and instance segmentation.

[0005] However, despite the good performance of multimodal image fusion technology in low-light environments, effectively aligning instances in visible light and infrared images, especially when there is heterogeneity between the images, remains a challenging research topic. Due to the differences in imaging principles and sensor characteristics between visible light and infrared images, their image content often deviates, leading to alignment errors in the fused image. Therefore, effectively aligning instances in both types of images while preserving image details and enhancing object information remains a key issue in the field of image segmentation. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by the present invention is that existing instance segmentation methods for visible and infrared images have deviations in the fusion process and cannot accurately identify and segment instance objects.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an instance segmentation method based on visible light and infrared image feature fusion, comprising:

[0009] In low-light environments, infrared and visible light images are acquired;

[0010] The visible light image is analyzed using the first model to obtain the instance segmentation result in the first state;

[0011] The infrared image is analyzed using the second model to obtain the instance segmentation result in the second state;

[0012] Align the instance segmentation results in the first and second states to adjust for heterogeneity between visible light and infrared images;

[0013] Based on the analysis results, the final instance segmentation results are generated.

[0014] As a preferred embodiment of the instance segmentation method based on visible light and infrared image feature fusion described in this invention, wherein: visible light and infrared images are captured simultaneously using a visible light and infrared image fusion device;

[0015] The infrared image is acquired by the infrared radiation image acquisition section;

[0016] The visible light image is acquired through the visible light image acquisition section.

[0017] As a preferred embodiment of the instance segmentation method based on visible light and infrared image feature fusion described in this invention, wherein: the first model is specifically an IAT-YOLO low-light detection model, comprising:

[0018] It consists of three parts: the backbone network, the neck network, and the detection head.

[0019] The IAT low-light enhancement module is used as the head of the backbone network, while the rest of the backbone network retains the original CSPDarkNet53 part.

[0020] Four attention modules are added between the backbone network and the neck network, and are connected sequentially through MP modules;

[0021] In each attention module group, insert a CA module before the ELAN module;

[0022] The neck network integrates the three-scale output characteristics of the second, third, and fourth ELAN modules in the backbone network; at the same time, sppcspc-r is introduced at the head of the neck network.

[0023] The detection head maps feature representations of different scales to the output layer through the classifier layer, thereby predicting the bounding box information of the segmented object and obtaining the instance segmentation result in the first state;

[0024] The first state includes a state where the image content is a visible light image;

[0025] Here, an instance represents an object to be identified in the image.

[0026] As a preferred embodiment of the instance segmentation method based on visible light and infrared image feature fusion described in this invention, the IAT low-light enhancement module consists of two branches, wherein:

[0027] The local branch is responsible for inverse mapping adjustments, used to capture local details and features in the image, and to perform local enhancement operations;

[0028] The global branch is responsible for learning ISP parameters, capturing global information and contextual relationships of the image, and performing global enhancement operations.

[0029] As a preferred embodiment of the instance segmentation method based on visible light and infrared image feature fusion described in this invention, the second model includes segmenting instances in the infrared image through a pre-trained neural network.

[0030] The second state includes a state where the screen content is an infrared image.

[0031] As a preferred embodiment of the instance segmentation method based on visible light and infrared image feature fusion described in this invention, the method of aligning the instance segmentation results in the first state and the second state includes fine-tuning the relative positions of the visible light and infrared images according to the weighting center of the instance segmentation results in each state, thereby achieving alignment of the instance segmentation results in the two states.

[0032] The fine-tuning process is as follows:

[0033] Step 1: Obtain the instance segmentation results in the first state and the second state respectively;

[0034] Step 2: Take any image in any state, locate the contours of the instance segmentation results in the image, and select the instances based on the location results;

[0035] If the contour of the same instance m can be detected in the segmentation results of both states, then instance m is selected in the segmentation results of both states respectively.

[0036] Step 3: Based on the contour positioning results of each selected instance, obtain the geometric centroid of each selected instance;

[0037]

[0038]

[0039] in, This represents the coordinates of the geometric centroid of instance m on the x-axis. The coordinates of the geometric centroid of instance m on the y-axis are given; n represents the total number of pixels in the instance's outline; i represents the pixel index, which is incremented along the outline starting from any pixel in the outline. Represents the x-coordinate of the i-th pixel of instance m; Represents the ordinate of the i-th pixel of instance m;

[0040] Step 4: In each state, assign weights to each selected instance;

[0041]

[0042] in, This indicates the weight assignment for instance m in state j; Indicates the standard weight value. This represents the area enclosed by the contour of instance m in state j; the value of j is 1 or 2, representing the first state and the second state respectively. This represents the adjustment coefficient, which is adjusted according to the changes in the instances in the image to ensure that the sum of the weights of all instances in the image is 1.

[0043] Step 5: Based on the weights, reconcile the weights in the two states so that the weight of instance m is the average of the weights in the two states.

[0044] in, This indicates the weight assignment for instance m;

[0045] Step 6: Align the selected instances according to their geometric centroid and weight.

[0046] The objective function is to minimize the weighted result of the centroid distance of the instances.

[0047]

[0048] in, This represents the geometric centroid of instance m after alignment in the first state; Represents Euclidean distance; This represents the weight of instance m; This represents the geometric centroid of instance m in the second state; M represents the number of selected instances.

[0049] Step 7, set In the first state, the geometric centroid of instance m before alignment. This represents the amount of image displacement during the alignment process of the visible light image. After aligning the image edges in the two states, the visible light image is moved by the amount of image displacement to complete the alignment of the instance segmentation results in the two states.

[0050] As a preferred embodiment of the instance segmentation method based on visible light and infrared image feature fusion described in this invention, the adjustment of the heterogeneity between visible light and infrared images includes fusing features of visible light and infrared images, and performing adversarial analysis on the instance segmentation results in the two states, and using the adversarial analysis results to generate the final instance segmentation result.

[0051] Adversarial analysis is performed using pre-trained generative adversarial networks;

[0052] Step 1: Obtain the aligned visible light and infrared images of the instance segmentation results. Assume that the aligned visible light and infrared images are combined into a stacked image, and assign two attributes to each pixel of the stacked image. ;

[0053] in, In the first state, the u-th pixel assigns an attribute to the stacked image, with a value of 0 or 1, where 0 indicates that there is no instance at the pixel position and 1 indicates that there is an instance at the pixel position. In the second state, the w-th pixel assigns an attribute to the stacked image, with a value of 0 or 1, where 0 indicates that there is no instance at the pixel position and 1 indicates that there is an instance at the pixel position. This represents the k-th pixel in the stacked image;

[0054] If a pixel in the stacked image has an attribute in only one state, then the attribute in the other state is padded with 0;

[0055] Step 2, filter all The pixels with both attributes set to 0 are used to obtain the portion of the stacked image that does not contain instances.

[0056] Step 3, filter all The portion of the stacked image containing instances is obtained by taking pixels where both attributes are 1.

[0057] Step 4: Map the portion of the stacked image that does not contain instances to the two original images to obtain mapped portion 1; map the portion of the stacked image that contains instances to the two original images to obtain mapped portion 2;

[0058] Step 5: Merge the mapping portions 1 and 2 of the two original images respectively to generate an image fusion result without instance portions and an image fusion result with instance portions in the stacked image; and in the fusion result, mark the location of each identified instance;

[0059] Step 6, filter all For pixels where two attributes are not both 1 or 0, the portion of the stacked image suspected of containing instances is obtained;

[0060] Step 7: Map the suspected instance portions in the stacked image to the two original images to obtain mapped portion 3; analyze each suspected instance portion in mapped portion 3 separately.

[0061] Calculate the sharpness of each suspected instance portion in the visible light image; if the sharpness is greater than a preset value D1, then the discriminator increases the confidence level of the visible light image in the generator based on the sharpness.

[0062] If the sharpness is less than the preset value D2, the discriminator will increase the confidence of the infrared image in the generator based on the sharpness.

[0063] If the resolution is not greater than the preset value D1 and not less than the preset value D2, then no adjustment will be made to the confrontation process;

[0064] When inputting into the generator, visible light and infrared light are used as different channels, and different channel attention weights are applied according to the confidence level.

[0065] Step 8: Based on the adversarial results, fuse the stacked images; simultaneously, mark the parts determined to contain instances.

[0066] The image fusion results obtained in steps 5 and 8 are stitched together to obtain the final instance segmentation result.

[0067] An instance segmentation system based on visible light and infrared image feature fusion using the method described in this invention, wherein:

[0068] The acquisition unit acquires infrared and visible light images in low-light environments;

[0069] The analysis unit uses a first model to analyze the visible light image and obtains the instance segmentation result in the first state; it uses a second model to analyze the infrared image and obtains the instance segmentation result in the second state.

[0070] The adjustment unit aligns the instance segmentation results in the first and second states and adjusts the heterogeneity between visible light and infrared images.

[0071] The generation unit generates the final instance segmentation result based on the analysis results.

[0072] A computer device includes: a memory and a processor; the memory stores a computer program, wherein: when the processor executes the computer program, it implements the steps of the method described in any one of the present invention.

[0073] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the present invention.

[0074] The beneficial effects of this invention are as follows: The instance segmentation method based on visible light and infrared image feature fusion provided by this invention significantly improves the accuracy and robustness of instance segmentation in low-light or weak-light environments by fusing visible light and infrared image features. By aligning and optimizing the instance segmentation results from the two images, it overcomes the shortcomings of traditional methods in low-light environments, providing more accurate object detection and segmentation. Simultaneously, combining image fusion with a generative adversarial network (GAN) effectively reduces alignment errors caused by heterogeneity, improves image quality and detail preservation, and provides reliable technical support for image analysis tasks in practical applications. Attached Figure Description

[0075] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 The overall flowchart of an instance segmentation method based on visible light and infrared image feature fusion provided in the first embodiment of the present invention;

[0077] Figure 2 A generator structure block diagram for an instance segmentation method based on visible light and infrared image feature fusion provided in the first embodiment of the present invention;

[0078] Figure 3 The first embodiment of the present invention provides a generator weight adjustment logic diagram for an instance segmentation method based on visible light and infrared image feature fusion. Detailed Implementation

[0079] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0080] Example 1, referring to Figure 1 , Figure 2 , Figure 3 As an embodiment of the present invention, an instance segmentation method based on visible light and infrared image feature fusion is provided, comprising:

[0081] S1: Acquire infrared and visible light images in low-light environments.

[0082] A visible light and infrared image fusion device is used to simultaneously capture images of both visible light and infrared light. The infrared image is acquired through an infrared radiation image acquisition section, while the visible light image is acquired through a visible light image acquisition section. By simultaneously capturing and fusing infrared and visible light images, image detail and feature recognition capabilities are enhanced, improving environmental perception and ensuring the accuracy of target detection, recognition, and analysis under low-light conditions.

[0083] The following are some common implementation methods for visible light and infrared image fusion devices:

[0084] 1. Dual-mode camera (visible light and infrared dual cameras):

[0085] Dual-modal imaging systems: These devices integrate visible light and infrared sensors, typically providing two different sensors within the same camera module: one for capturing visible light images and the other for capturing infrared images. They generally ensure that both images are captured from the same viewpoint and at the same time, resulting in high alignment accuracy and facilitating subsequent fusion processing.

[0086] Typical equipment, such as FLIR's dual-mode infrared thermal imager, combines infrared and visible light images in the same device, providing synchronized image data for monitoring and analysis. Registration of the infrared and visible light images is crucial in these devices; therefore, they typically feature high-precision synchronization capabilities.

[0087] 2. Multi-sensor camera system:

[0088] Visible light + infrared camera array: This system consists of two independent sensors (one for visible light and one for infrared), typically fixed in place by a precise mount to ensure that the viewing angles and positions of the two sensors are perfectly aligned. This device simultaneously captures visible light and infrared images of the same scene, and then performs image alignment and fusion at the back end.

[0089] Features: These multi-sensor camera systems can use the same shooting angle, focal length, and resolution, thereby reducing fusion problems caused by differences in perspective.

[0090] 3. Integrated optical and thermal imaging system:

[0091] This device combines visible light imaging and infrared thermal imaging in the same system and is commonly used in military, security, and drone applications. The optical lens and the infrared thermal imaging lens are aligned using the same external frame, ensuring that both images are acquired from the same perspective.

[0092] Typical applications include security monitoring systems, drones, and satellite systems, which are designed with the fusion of two types of images in mind. They can acquire high-quality visible light and infrared images at the same time and then fuse them using subsequent algorithms.

[0093] In this embodiment, a dual-modal camera is used to complete the data acquisition.

[0094] If the two acquisition devices are not integrated on the same camera, when visually adjusting the lens of the infrared radiation image acquisition part or the visible light image acquisition part, the other lens is visually adjusted synchronously through a following mechanism; when adjusting one camera, the other camera is adjusted at the same time so that the center of the infrared image and the center of the visible light image are aligned.

[0095] S2: Analyze the visible light image using the first model to obtain the instance segmentation result in the first state.

[0096] The first model is specifically the IAT-YOLO low-light detection model, comprising three parts: a backbone network, a neck network, and a detection head. The IAT low-light enhancement module serves as the head of the backbone network, while the remaining parts of the backbone network retain the original CSPDarkNet53 portion. Four attention modules are added between the backbone network and the neck network, connected sequentially via MP modules. In each attention module, a CA module is inserted before the ELAN module. The neck network integrates the three-scale output characteristics of the second, third, and fourth ELAN modules in the backbone network; simultaneously, sppcspc-r is introduced at the head of the neck network.

[0097] It's worth noting that Continuous Attention (CA) is an advanced attention mechanism designed to improve the performance of convolutional neural networks in image processing tasks by leveraging spatial relationships within feature maps. Unlike traditional global average pooling, CA decomposes the pooling operation along the vertical axis into two consecutive directions, preserving spatial information in each direction and generating two independent feature maps. This captures long-range dependencies and enhances the ability to capture details. By incorporating CA into the backbone network, the location and number of pigs can be accurately identified by focusing on the resulting global location, thereby suppressing enhanced image noise and improving the model's perception of pigs. Compared to traditional attention mechanisms, CA exhibits higher sensitivity and accuracy in processing detailed features, significantly improving the model's performance in tasks requiring precise spatial localization, such as object detection and image segmentation.

[0098] The sppcspc-r module integrates information at different characteristic scales through four different max-pooling operations (13x13, 9x9, 5x5, and 1x1 pooling kernels) to reduce computation and improve processing speed. Since extreme pixel values ​​may exist when processing pig images in low-light environments, the original SiU activation function may suffer from gradient vanishing problems. Therefore, SiLU is replaced by ReLU. ReLU exhibits good computational properties during neural network training and can effectively alleviate gradient vanishing issues.

[0099] The detection head maps feature representations at different scales to the output layer through a classifier layer, thereby predicting the bounding box information of the segmented object and obtaining the instance segmentation result in the first state. The first state includes a state where the image content is a visible light image. Here, an instance represents the object to be identified in the image.

[0100] In recent years, image enhancement techniques have been employed to address the challenge of target detection in low-light environments. This method is not only low-cost but also possesses broad virtual adaptability and high flexibility. However, its potential negative impacts cannot be ignored. To visually demonstrate the impact of image enhancement algorithms on target detection performance and to select the most suitable algorithm for a low-light target detection network, we combined the aforementioned five image enhancement algorithms with the YOLOv7 model for training and detection. The experimental results are shown in Table 1, illustrating the impact of different image enhancement algorithms on target detection under low-light conditions.

[0101] Table 1 Experimental Results

[0102]

[0103] We can observe that the IAT image enhancement algorithm has a small number of parameters and performs best when combined with the YOLOv7 model. Therefore, IAT is chosen as the image enhancement module.

[0104] The Illumination Adaptive Transformer (IAT) is a lightweight network specifically designed for real-time time-light enhancement and exposure correction. It improves the enhancement effect and visual quality of flowing light images by adaptively adjusting the brightness and contrast of the image, as well as through multi-scale feature extraction and global contextual information enhancement. The IAT low-light enhancement module consists of two branches, wherein:

[0105] The local branch, responsible for inverse mapping adjustments, captures local details and features in the image and performs local enhancement operations. This branch predicts pixel-level multiplication and addition mappings, mapping the sRGB image back to the original RGB space, and dynamically adjusts the image brightness by tweaking key parameters in the ISP process. Furthermore, the local branch replaces the attention module in the Transformer with depthwise convolution, achieving a lightweight design.

[0106] Global Branch: Responsible for learning ISP parameters, capturing global information and contextual relationships within the image, and performing global enhancement operations. This branch uses an attention mechanism to predict the color matrix and gamma values ​​that control the global information of the image, and achieves adaptive adjustment through a dynamic query learning strategy.

[0107] S3: Analyze the infrared image using the second model to obtain the instance segmentation result in the second state.

[0108] The second model includes segmenting instances in the infrared image using a pre-trained neural network. The second state includes a state where the image content is an infrared image.

[0109] In this embodiment, the second model can be U-Net. Since infrared images typically have low contrast and blurry boundaries, U-Net's skip connections can help the model recover important information from details extracted from lower layers, making it particularly suitable for tasks requiring fine segmentation.

[0110] In other feasible embodiments, Mask R-CNN can also be used. For instance segmentation tasks in infrared images, Mask R-CNN can not only identify objects but also accurately segment their contours, adapting to the challenges of low contrast and blurred details. It can optimize segmentation results by adding Region-of-Interest (ROI) Alignment and Non-Maximum Suppression (NMS).

[0111] S4: Align the instance segmentation results in the first and second states to adjust for heterogeneity between visible light and infrared images.

[0112] Aligning the instance segmentation results in the first and second states includes fine-tuning the relative positions of the visible light and infrared images based on the weighting center of the instance segmentation results in each state, thereby achieving alignment of the instance segmentation results in the two states.

[0113] The fine-tuning process is as follows:

[0114] Step 1: Obtain the instance segmentation results in the first state and the second state respectively.

[0115] Step 2: Take any image in any state, locate the outline of the instance segmentation result in the image, and select the instance based on the location result.

[0116] If the contour of the same instance m can be detected in the segmentation results of both states, then instance m is selected in the segmentation results of both states respectively. (The determination of the same instance can be made from multiple aspects: if instance m is a moving object, it can be determined whether it is the same instance by analyzing the consistency of the action. If it is not a moving object, it can be determined based on the similarity of the instances and their positions in the image. Because the visible light and infrared image fusion device needs to ensure the consistency of the image during configuration, the strategy in this step is only a fine adjustment of the image alignment. The position of the same object in the two images will not differ too much. If the relative distance of the instance contour deviates from a certain threshold, as long as the contour similarity is greater than the preset value, it can still be determined to be the same instance. If the instance contour in any image is determined to be an instance that mixes multiple objects in another image, it will not be selected due to the large noise in the image, which can ensure the robustness of alignment.)

[0117] Step 3: Based on the contour positioning results of each selected instance, obtain the geometric centroid of each selected instance.

[0118]

[0119]

[0120] in, This represents the coordinates of the geometric centroid of instance m on the x-axis. The coordinates of the geometric centroid of instance m on the y-axis are given; n represents the total number of pixels in the instance's outline; i represents the pixel index, which is incremented along the outline starting from any pixel in the outline. Represents the x-coordinate of the i-th pixel of instance m; This represents the ordinate of the i-th pixel of instance m.

[0121] Step 4: In each state, assign weights to each selected instance.

[0122]

[0123] in, This indicates the weight assignment for instance m in state j; Indicates the standard weight value. This represents the area enclosed by the contour of instance m in state j; the value of j is 1 or 2, representing the first state and the second state respectively. This represents the adjustment coefficient, which is adjusted according to the changes in the instances in the image to ensure that the sum of the weights of all instances in the image is 1.

[0124] Step 5: Reconcile the weights in the two states. .

[0125] in, This indicates the weight assignment for instance m.

[0126] Step 6: Align the selected instances according to their geometric centroid and weight.

[0127] The objective function is to minimize the weighted result of the centroid distance of the instances.

[0128]

[0129] in, This represents the geometric centroid of instance m after alignment in the first state; Represents Euclidean distance; This represents the weight of instance m; This represents the geometric centroid of instance m in the second state; M represents the number of selected instances.

[0130] Step 7, set In the first state, the geometric centroid of instance m before alignment. This represents the amount of image displacement during the alignment process of the visible light image. After aligning the image edges in the two states, the visible light image is moved by the amount of image displacement to complete the alignment of the instance segmentation results in the two states.

[0131] It's important to understand that precise alignment of instance segmentation results across the two states is achieved through fine-tuning and displacement of the visible light image. This step first requires calculating the geometric centroid of each instance and optimizing it based on its weighted result. Then, using the weighted result of the instance centroid distance as the objective function, the relative displacement between instances in the two states is calculated. By minimizing this objective function, the instance centroids in the two states can be made as close as possible, ensuring accurate instance alignment.

[0132] Furthermore, due to the heterogeneity of visible light and infrared images in terms of imaging principles and sensor differences, directly fusing or merging image segmentation results may lead to alignment errors. This step reduces these errors by fine-tuning the displacement of the visible light image. By fine-tuning the image based on instance centroids and weighting values, the spatial relationships of instances in the two images can be correctly aligned, especially when objects are moving or their positions are slightly shifted. This method makes the alignment and fusion of multiple instances in an image more stable and accurate. During the fine-tuning process, the calculated displacement is applied to the visible light image to achieve the final alignment. This process aligns the centroids of instances in both states, ultimately resulting in more accurate and coherent instance segmentation results, particularly in cross-modal image fusion (such as visible light and infrared images), ensuring the consistency and accuracy of image segmentation results.

[0133] The visible light and infrared images are fused for features, and the instance segmentation results in the two states are subjected to adversarial testing. The final instance segmentation result is generated using the adversarial results.

[0134] Adversarial analysis is performed using pre-trained generative adversarial networks.

[0135] Step 1: Obtain the aligned visible light and infrared images of the instance segmentation results. Assume that the aligned visible light and infrared images are combined into a stacked image, and assign two attributes to each pixel of the stacked image. .

[0136] in, In the first state, the u-th pixel assigns an attribute to the stacked image, with a value of 0 or 1, where 0 indicates that there is no instance at the pixel position and 1 indicates that there is an instance at the pixel position. In the second state, the w-th pixel assigns an attribute to the stacked image, with a value of 0 or 1, where 0 indicates that there is no instance at the pixel position and 1 indicates that there is an instance at the pixel position. This represents the k-th pixel in the stacked image.

[0137] If a pixel in the stacked image has an attribute in only one state, then the attribute in the other state is padded with 0.

[0138] The judgment in step 1 above is not absolute. In fact, the judgment of each pixel is based on the overall macroscopic representation. For example, (in an image) if a pixel in a certain region is 0, but all the surrounding pixels are 1, then the pixel that is 0 will be assimilated and also be treated as 1.

[0139] Step 2, filter all The pixels with both attributes set to 0 are used to obtain the portion of the stacked image that does not contain instances.

[0140] Step 3, filter all The portion of the stacked image containing instances is obtained by identifying pixels where both attributes are 1.

[0141] Step 4: Map the portion of the stacked image that does not contain instances to the two original images to obtain mapped portion 1; map the portion of the stacked image that contains instances to the two original images to obtain mapped portion 2 (the identification of this portion is not absolute; based on the individual size of the instance object, find a region of roughly the same size in the stacked image to ensure that the vast majority of points in the region are simultaneously 1).

[0142] Step 5: The mapping portions 1 and 2 of the two original images are fused to generate an image fusion result without instance portions and an image fusion result with instance portions in the stacked image; and in the fusion result, the location of each identified instance is marked.

[0143] Step 6, filter all Pixels whose two attributes are not both 1 or 0 simultaneously are identified as the parts of the stacked image that are suspected of containing instances. (This identification is not absolute; it also considers the individual size of the instance object and finds regions of roughly the same size in the stacked image, ensuring that the vast majority of pixels in the region are not both 1 or 0. This usually occurs when an instance exists in one image but cannot be identified in another.)

[0144] Step 7: Map the suspected instance portion in the stacked image to the two original images to obtain mapping portion 3; analyze each suspected instance portion in mapping portion 3 separately.

[0145] Calculate the sharpness of each suspected instance portion in the visible light image; if the sharpness is greater than a preset value D1, then the discriminator increases the confidence level of the visible light image in the generator based on the sharpness; the increase ratio is expressed as:

[0146]

[0147] in, This indicates the percentage increase in the credibility of the visible light image in the discriminator during the adversarial process; Indicates parameter adjustment; This indicates the sharpness of instance t in the visible light image.

[0148] If the sharpness is less than the preset value D2, the discriminator increases the confidence level of the infrared image in the generator based on the sharpness; the increase ratio is expressed as:

[0149]

[0150] Among them, the preset value D2 is less than the preset value D1; This indicates the percentage increase in the credibility of infrared images in the discriminator during the confrontation process; Indicates parameter adjustment; This indicates the sharpness of instance t in the visible light image.

[0151] If the resolution is not greater than the preset value D1 and not less than the preset value D2, then no adjustment will be made to the anti-countermeasure process.

[0152] When inputting into the generator, visible light and infrared light are treated as different channels, and different channel attention weights are applied based on the confidence level.

[0153] Step 8: Based on the adversarial results, fuse the stacked images; at the same time, mark the parts that are determined to exist as instances.

[0154] The image fusion results obtained in steps 5 and 8 are stitched together to obtain the final instance segmentation result.

[0155] It's worth noting that step 5 involves direct fusion of either the pure background portion or the portion containing instances. Because the content of these two portions is clear and highly directional, fusion doesn't require extensive computation; it can be accomplished primarily through efficient fusion methods. Since the region content is well-defined, efficient fusion methods (such as simple weighted average, Laplacian pyramid, U-Net generator, etc.) can be directly used, avoiding complex deep feature calculations and improving fusion efficiency.

[0156] Step 8 involves identifying portions of an instance object that exists in one image but not in another. This step deals with instances that are present in one image but missing in another. By specifically identifying and fusing only these instances, unnecessary computations can be reduced, particularly avoiding redundant computations on objects present in both images. By focusing on instances that appear in only one image, redundant computations in regions present in both images can be avoided. For example, if an instance exists in a visible light image but not in an infrared image, a Generative Adversarial Network (GAN) can optimize only for the instance in the visible light image, without needing to perform additional computations on the infrared image. These instances often have unique features that are difficult to identify in another image. By concentrating computational resources on these challenging instances, their segmentation and fusion results can be optimized more efficiently, rather than wasting computation on simple instances present in both images.

[0157] By breaking down the computational task into several stages (e.g., processing the background first, then common instances, and finally unique instances), the computational load at each step can be effectively reduced. Focusing on only specific types of instances at each stage avoids redundant computation across all regions.

[0158] Additionally, it's worth mentioning that the adversarial process involves using a Generative Adversarial Network (GAN) to optimize the fusion result for each different part of the image. Through adversarial training, the generator learns how to generate a fused image for each part based on the input visible light and infrared images (in reality, the size of each part is roughly the size of the target instance, so each part is generally an independent region), and labels the location to indicate whether an instance exists. Simultaneously, a discriminator judges the realism of the fused image and its labels, thereby improving the final instance segmentation quality.

[0159] The generator's task is to produce a fused image based on the input visible light and infrared images. For each part (background, common instance, unique instance), the generator needs to synthesize the image according to a specific target.

[0160] The input consists of two channels: a visible light image and an infrared image (the analysis is performed on each part of the region; therefore, the fusion of these two channels is also performed on this part).

[0161] Output: The main branch generates the fused image (for the part being analyzed), and the instance branches generate the location markers for each instance.

[0162] like Figure 2 As shown, this U-Net generator is used for visible-infrared image fusion tasks. Input: Visible / infrared image. Features are extracted by the encoder (downsampling). Fusion is performed through skip connections and the decoder (upsampling). Output: High-quality fused image. This network belongs to an encoder-decoder structure, including: Encoding (downsampling) part (blue part on the left): Features are extracted through convolution (Conv3×3). Downsampling is performed through max pooling (MaxPool2×2) to reduce spatial size and improve semantic information. Bottleneck layer (bottom): Deep features are extracted through maximum compression. A 1024-channel convolutional layer is used here. Decoding (upsampling) part (orange part on the right): Upsampling is performed using bilinear interpolation. Features from the encoding stage are concatenated back through skip connections to preserve high-resolution information. Output layer: The final output image or classification probability map is generated through 1×1 convolution (Conv1×1).

[0163] Detailed implementation steps:

[0164] (1) Input layer:

[0165] Input: 480×480×34 (RGB image).

[0166] After convolution with 3×3+BN+ReLU, a 64-channel feature map is output.

[0167] (2) Encoding part (Encoder, downsampling):

[0168] Level 1:

[0169] 480×480×64→MaxPool2×2→240×240×64.

[0170] Level 2:

[0171] 240×240×128→MaxPool2×2→120×120×128.

[0172] Level 3:

[0173] 120×120×256→MaxPool2×2→60×60×256.

[0174] Level 4:

[0175] 60×60×512→MaxPool2×2→30×30×512.

[0176] Layer 5 (Bottleneck Layer):

[0177] 30×30×1024.

[0178] (3) Decoding section (Decoder, upsampling):

[0179] Level 1:

[0180] 30×30×10243→Bilinear Interpolate→60×60×512.

[0181] It is spliced ​​with the 512-channel features of the encoder (skip connection).

[0182] Level 2:

[0183] 60×60×512→Bilinear Interpolate→120×120×256.

[0184] It is spliced ​​with the 256-channel features of the encoder.

[0185] Level 3:

[0186] 120×120×256→Bilinear Interpolate→240×240×128.

[0187] It is spliced ​​with the 128-channel features of the encoder.

[0188] Level 4:

[0189] 240×240×128→Bilinear Interpolate→480×480×64.

[0190] It is spliced ​​with the 64-channel features of the encoder.

[0191] (4) Output layer:

[0192] After a 1×1 convolution, the final output is a 2-channel image (which may be a mask or a segmentation result).

[0193] Here, "Conv3×3" represents a 3×3 convolutional kernel, i.e., a 3×3 filter used for calculating image features in a sliding window. "BN" represents batch normalization. "ReLU" represents a non-linear activation function. MaxPool2×2 represents pooling. Bilinear Interpolate is an image upsampling / downsampling method commonly used in deep learning, computer vision, and image processing tasks, especially in the upsampling process of convolutional neural networks (CNNs).

[0194] During the training phase, analysis is required based on the input visible light image, infrared image, and fused image. The realism of the fused image needs to be evaluated based on feedback from the discriminator. During recognition, it can be used for online target detection to ensure sufficiently high confidence in the target region. The weights of visible light / infrared light are dynamically adjusted based on the discrimination results, optimizing the fusion effect's weights on each channel of the generator.

[0195] During the inference phase, if the discriminator deems the infrared information insufficient, it increases the weight of the infrared channel; if the discriminator deems visible light details lost, it increases the weight of the visible light channel. Let the weights output by the discriminator during the training phase be p1 and p2; p1 represents the weight of the visible light image as input to the generator, and p2 represents the weight of the infrared image as input to the generator.

[0196] It is important to note that when the generator's weights are not adjusted, the fusion result P should be a weighted result of the two fixed weights.

[0197]

[0198] in, This represents the contents of the visible light image channel; This indicates the contents of the infrared image channel; and The sum is 1.

[0199] like Figure 3 When the generator's weights are adjusted, the final fusion result P is obtained by weighting two variable weights.

[0200]

[0201] Sharpness in visible light images; if the sharpness is greater than the preset value D1, then ; If the resolution is less than the preset value D2, then ; Since these instances typically exist only in one image, their features differ significantly between visible and infrared images, potentially resulting in lower sharpness and reliability. To better handle these special regions, variable weights are used to adjust the contribution ratio of the image based on sharpness, enabling the system to flexibly address these challenging instances.

[0202] It's worth noting that PatchGAN (Local Discriminator) is a discriminator architecture based on a convolutional neural network (CNN), primarily used for generative adversarial network (GAN) tasks, particularly suitable for image generation and transformation (such as infrared-visible image fusion, image inpainting, super-resolution, etc.). Compared to a global discriminator, PatchGAN focuses only on local regions of an image, enabling it to evaluate image quality more precisely.

[0203] In one feasible implementation, PatchGAN mainly consists of a 5-layer convolutional network, each layer using 3×3 convolutions with a stride of 2, and reducing the feature map size. The structure is shown in Table 2.

[0204] Table 2 Network Structure Table

[0205]

[0206] S5: Based on the analysis results, generate the final instance segmentation results.

[0207] The image fusion results obtained from steps 5 and 8 in S4 are stitched together to obtain the final instance segmentation result.

[0208] In the background obtained in step 5, the image fusion results of the instance obtained in step 5 and each part obtained in step 8 are stitched together. Let the centroid of the contour of the e-th image fusion result (assumed to be instance m) be... ;Will Set in stacked images and The midpoint of the connecting line is used to stitch together the image fusion results. If only... If it exists, then directly... Set in existence Location; Similarly.

[0209] In addition, to avoid obvious seams between different regions, smooth transition techniques (such as weighted averaging, gradient blending, etc.) can be used to ensure smooth transitions between region boundaries. For regions with shared instances and unique instances, edge smoothing techniques can be applied to reduce hard boundaries during merging and ensure a natural and smooth image.

[0210] After alignment and smooth transition processing, pixel-level stitching is used to merge the images of each region together.

[0211] For background regions, shared instance regions, and unique instance regions, each has its own image output. During stitching, the pixel values ​​of each part are merged into the same image. During the stitching process, blank or overlapping areas may appear due to image alignment issues. For blank areas, appropriate background information may need to be filled in to avoid generating blank areas; for overlapping areas, weighted or blending methods are needed to ensure that no duplicate instances occur.

[0212] Finally, the stitched image contains all the information about the background, common instances, and unique instances, generating a complete instance segmentation result image.

[0213] The image should clearly display the object instance and ensure seamless transitions between different areas, with a smooth and natural blending of the background and instance.

[0214] On the other hand, this embodiment also provides an instance segmentation system based on visible light and infrared image feature fusion, which includes:

[0215] The acquisition unit acquires infrared and visible light images in low-light environments.

[0216] The analysis unit uses a first model to analyze the visible light image and obtains the instance segmentation result in the first state; it uses a second model to analyze the infrared image and obtains the instance segmentation result in the second state.

[0217] The adjustment unit aligns the instance segmentation results in the first and second states and adjusts the heterogeneity between visible light and infrared images.

[0218] The generation unit generates the final instance segmentation result based on the analysis results.

[0219] If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0220] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0221] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0222] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0223] Example 2 is an embodiment of the present invention, which provides an instance segmentation method based on visible light infrared image feature fusion. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0224] The experiment will collect visible light and infrared images under low light and weak light environments, and use the fusion method of this invention to segment black pig instances to verify the high segmentation accuracy and stability of the method under different environments. The results are shown in Table 3.

[0225] Scene selection: A typical farm environment was chosen, containing multiple black pigs, with a background including grassland and fences. The experimental environment features low-light and dim-light conditions (e.g., dusk or night). Due to uneven lighting in the environment, the contrast between the target object (black pigs) and the background is low, making the instance segmentation task relatively complex.

[0226] Equipment Configuration: The image acquisition equipment used in the experiment is a dual-mode image acquisition system, including a visible light image sensor (RGB) and an infrared image sensor. The equipment has a resolution of 1920x1080 and can simultaneously acquire two types of images. The equipment has high alignment accuracy to ensure the synchronous acquisition and alignment of visible light and infrared images.

[0227] Experimental objective: To verify the accuracy, stability, and robustness of the instance segmentation method based on the fusion of visible light and infrared image features in low-light and weak-light environments for black pig instance segmentation, and to verify the effectiveness of the method through actual observation and data analysis.

[0228] Experimental steps:

[0229] Step 1: Use the IAT-YOLO low-light detection model to perform instance segmentation on the visible light image to obtain the instance segmentation results in the first state.

[0230] Step 2: Use a pre-trained neural network to perform instance segmentation on the infrared image to obtain the instance segmentation results in the second state.

[0231] Step 3: Optimize the instance segmentation results of visible light and infrared images using a generative adversarial network (GAN), perform image fusion and instance alignment, and obtain the final instance segmentation results.

[0232] Step 4: Evaluate the segmentation effect by measuring indicators such as segmentation accuracy, image quality, and processing time, and compare it with manually labeled segmentation results to verify the reliability and accuracy of the method.

[0233] Evaluation indicators:

[0234] The criteria for judging the accuracy of instance segmentation are: precision, recall, and F1 score.

[0235] Processing time: The time to process each image (in seconds), used to evaluate the real-time performance of the method.

[0236] Image quality: The structural similarity index (SSIM) is used to evaluate the quality of the fused images.

[0237] Manual annotation comparison: Compare the segmentation results with those of manual annotation to evaluate the accuracy of segmentation.

[0238] Table 3 Experimental Results

[0239]

[0240] The experimental data show that the proposed method exhibits high segmentation accuracy and stability under various environments. In particular, under low light and weak light conditions, the fusion method significantly improves segmentation accuracy compared to a single image.

[0241] In low-light environments (Experiment 1), the segmentation accuracy using visible light images alone was 87.3%, while the segmentation accuracy using infrared images was 83.2%. Clearly, the segmentation performance of both decreased under low-light conditions. However, the segmentation accuracy of the fused image reached 95.7%, an improvement of 9.7% compared to the single image.

[0242] In low-light environments (Experiment 2), the segmentation accuracy using visible light images alone was 88.1%, while that using infrared images was 85.5%. However, the segmentation accuracy of the fused image improved to 96.2%, an increase of 9.1%. This indicates that infrared images significantly complement visible light images, especially under low-light or weak-light conditions, and can greatly improve segmentation accuracy.

[0243] In nighttime environments (Experiment 4), although the accuracy of using visible light images and infrared images alone is low, the accuracy of the fusion method is 94.5%, an improvement of 10.0%. This shows the advantage of infrared images at night, especially their significant supplementary role in low-light environments.

[0244] The SSIM (Structural Similarity Index) value was used to evaluate the quality of the fused image. A higher SSIM value for the fused image compared to a single image indicates that the fused image not only improves segmentation accuracy but also demonstrates superior image quality. For example, in Experiment 1 (low-light environment), the SSIM value of the fused image was 0.92, while the visible light image was 0.87 and the infrared image was 0.83. The fused image exhibited better quality and a more natural transition between the background and the instance.

[0245] Although the fusion method involves training a generative adversarial network (GAN) on visible light and infrared images, the processing time remains between 5.1 and 5.6 seconds, meeting the requirements for real-time processing. This demonstrates the superiority of the method in terms of real-time performance, especially in low-light and weak-light environments, where the processing speed still meets the needs of practical applications.

[0246] The fusion effect was significantly improved in all experiments. The image fusion effect was significantly improved in low light and low light environments, especially in low light environment (experiment 1) and nighttime environment (experiment 4), the accuracy of the fused image was improved by 9.7% and 10.0% respectively, demonstrating that infrared images effectively complement visible light images. Especially under the condition that object instances are difficult to segment, infrared images provide additional recognition information for visible light images.

[0247] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An instance segmentation method based on visible light and infrared image feature fusion, characterized in that, include: In low-light environments, infrared and visible light images are acquired; The visible light image is analyzed using the first model to obtain the instance segmentation result in the first state; The infrared image is analyzed using the second model to obtain the instance segmentation result in the second state; Align the instance segmentation results in the first and second states to adjust for heterogeneity between visible light and infrared images; Based on the analysis results, the final instance segmentation results are generated; Aligning the instance segmentation results in the first state and the second state includes fine-tuning the relative positions of the visible light and infrared images based on the weighting center of the instance segmentation results in each state, thereby achieving alignment of the instance segmentation results in the two states. The fine-tuning process is as follows: Step 1: Obtain the instance segmentation results in the first state and the second state respectively; Step 2: Take any image in any state, locate the contours of the instance segmentation results in the image, and select the instances based on the location results; If the contour of the same instance m can be detected in the segmentation results of both states, then instance m is selected in the segmentation results of both states respectively. Step 3: Based on the contour positioning results of each selected instance, obtain the geometric centroid of each selected instance; Step 4: In each state, assign weights to each selected instance; Step 5: Based on the weights, reconcile the weights in the two states so that the weight of instance m is the average of the weights in the two states. Step 6: Align the selected instances according to their geometric centroid and weight. The objective function is to minimize the weighted sum of the centroid distances of instances. Step 7: Calculate the image displacement of the visible light image during the alignment process; after aligning the image edges in the two states, move the visible light image with the image displacement to complete the alignment of the instance segmentation results in the two states.

2. The instance segmentation method based on visible light and infrared image feature fusion as described in claim 1, characterized in that: Visible light and infrared image fusion equipment is used to capture images of both visible light and infrared simultaneously.

3. The instance segmentation method based on visible light and infrared image feature fusion as described in claim 2, characterized in that: The first model is specifically the IAT-YOLO low-light detection model, which includes: It consists of three parts: the backbone network, the neck network, and the detection head. The IAT low-light enhancement module is used as the head of the backbone network, while the rest of the backbone network retains the original CSPDarkNet53 part. Four attention modules are added between the backbone network and the neck network, and are connected sequentially through MP modules; In each attention module group, insert a CA module before the ELAN module; The neck network integrates the three-scale output characteristics of the second, third, and fourth ELAN modules in the backbone network; at the same time, sppcspc-r is introduced into the head of the neck network; sppcspc-r is based on the structure of sppcspc and fuses information of different characteristic scales through four different scales of max pooling operations. The detection head maps feature representations of different scales to the output layer through the classifier layer, thereby predicting the bounding box information of the segmented object and obtaining the instance segmentation result in the first state; The first state includes a state where the image content is a visible light image; Here, an instance represents an object to be identified in the image.

4. The instance segmentation method based on visible light and infrared image feature fusion as described in claim 3, characterized in that: The IAT low-light enhancement module consists of two branches, wherein: The local branch is responsible for inverse mapping adjustments, used to capture local details and features in the image, and to perform local enhancement operations; The global branch is responsible for learning ISP parameters, capturing global information and contextual relationships of the image, and performing global enhancement operations.

5. The instance segmentation method based on visible light and infrared image feature fusion as described in claim 4, characterized in that: The second model includes segmenting instances in the infrared image using a pre-trained neural network; The second state includes a state where the screen content is an infrared image.

6. The instance segmentation method based on visible light and infrared image feature fusion as described in claim 5, characterized in that: The adjustment of the heterogeneity between visible light and infrared images includes fusing features of visible light and infrared images, adversarializing the instance segmentation results in the two states, and using the adversarial results to generate the final instance segmentation result. Adversarial analysis is performed using pre-trained generative adversarial networks; Step 1: Obtain the aligned visible light and infrared images of the instance segmentation results. Assume that the aligned visible light and infrared images are combined into a stacked image, and assign two attributes to each pixel of the stacked image. ; in, In the first state, the u-th pixel assigns an attribute to the stacked image, with a value of 0 or 1, where 0 indicates that there is no instance at the pixel position and 1 indicates that there is an instance at the pixel position. In the second state, the w-th pixel assigns an attribute to the stacked image, with a value of 0 or 1, where 0 indicates that there is no instance at the pixel position and 1 indicates that there is an instance at the pixel position. This represents the k-th pixel in the stacked image; If a pixel in the stacked image has an attribute in only one state, then the attribute in the other state is padded with 0; Step 2, filter all The pixels with both attributes set to 0 are used to obtain the portion of the stacked image that does not contain instances. Step 3, filter all The portion of the stacked image containing instances is obtained by taking pixels where both attributes are 1. Step 4: Map the portion of the stacked image that does not contain instances to the two original images to obtain mapped portion 1; map the portion of the stacked image that contains instances to the two original images to obtain mapped portion 2; Step 5: Merge the mapping portions 1 and 2 of the two original images respectively to generate an image fusion result without instance portions and an image fusion result with instance portions in the stacked image; and in the fusion result, mark the location of each identified instance; Step 6, filter all For pixels where two attributes are not both 1 or 0, the portion of the stacked image suspected of containing instances is obtained; Step 7: Map the suspected instance portions in the stacked image to the two original images to obtain mapped portion 3; analyze each suspected instance portion in mapped portion 3 separately. Calculate the sharpness of each suspected instance portion in the visible light image; if the sharpness is greater than a preset value D1, then the discriminator increases the confidence level of the visible light image in the generator based on the sharpness. If the sharpness is less than the preset value D2, the discriminator will increase the confidence of the infrared image in the generator based on the sharpness. If the resolution is not greater than the preset value D1 and not less than the preset value D2, then no adjustment will be made to the confrontation process; When inputting into the generator, visible light and infrared light are used as different channels, and different channel attention weights are applied according to the confidence level. Step 8: Based on the adversarial results, fuse the stacked images; simultaneously, mark the parts determined to contain instances. The image fusion results obtained in steps 5 and 8 are stitched together to obtain the final instance segmentation result.

7. An instance segmentation system based on visible light and infrared image feature fusion using the method described in any one of claims 1-6, characterized in that: The acquisition unit acquires infrared and visible light images in low-light environments; The analysis unit uses a first model to analyze the visible light image and obtains the instance segmentation result in the first state; The infrared image is analyzed using the second model to obtain the instance segmentation result in the second state; The adjustment unit aligns the instance segmentation results in the first and second states and adjusts the heterogeneity between visible light and infrared images. The generation unit generates the final instance segmentation result based on the analysis results.

8. A computer device, comprising: Memory and processor; The memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images