A night vision fused image comprehensive quality evaluation method, medium and system

Abstract

The application provides a night vision fusion image comprehensive quality evaluation method, medium and system, belongs to image quality evaluation technical field, this night vision fusion image comprehensive quality evaluation method, medium and system include: the infrared and visible light color night vision fusion image in image database is pretreated, and the thermal target in image is eliminated;The image after pretreatment is divided into training set and test set;Support vector regressor is constructed, which is used for image feature extraction and fusion, the support vector regressor includes: image color feature extraction module, definition feature extraction module, deep feature extraction module and feature fusion module;The training set is input into the support vector regressor for training;The image to be evaluated is input into the support vector regressor after training, and the quality score of the input image is obtained.The application is sensitive to image color distortion, fully considers human eye visual characteristics, and can give evaluation results consistent with human eye subjective feeling.

Description

Technical Field

[0001] This invention belongs to the field of image quality evaluation technology, and specifically relates to a method, medium and system for comprehensive quality evaluation of night vision fusion images. Background Technology

[0002] Visible light (low-light) images contain rich scene details and texture information; infrared images can reflect the temperature information of objects and are less affected by weather and lighting conditions. Visible light (low-light) and infrared color night vision fusion technology combines the imaging advantages of both bands to produce pseudo-color fused images that highlight thermal targets, are rich in detail, and are suitable for human visual observation. Different color fusion algorithms produce drastically different visual effects. How to objectively evaluate the overall quality of fused images for specific visual tasks remains a challenge, both domestically and internationally, lacking a universally accepted theory and method. The research difficulty lies in the lack of ideal reference images for evaluating color fused images of low signal-to-noise ratio monochromatic weak visible light and monochromatic thermal radiation images invisible to the human eye. Furthermore, color fused images have unique image characteristics and specific application requirements, making them unsuitable for direct measurement using existing true-color image quality evaluation methods. Therefore, evaluating the ability of color fused images to meet specific application needs without a reference has become one of the key issues that urgently needs to be addressed in theoretical research and system applications both domestically and internationally.

[0003] Traditional methods rely on the brightness and contrast of images, pay limited attention to pixel relationships, are insensitive to changes in image color, and have technical problems in rarely considering the characteristics of human visual perception when evaluating image quality, resulting in low accuracy in scoring image quality. Summary of the Invention

[0004] In view of this, the present invention provides a method, medium and system for comprehensive quality evaluation of night vision fusion images, which can solve the technical problems of insensitivity to changes in image color and insufficient consideration of human visual characteristics when evaluating image quality.

[0005] This invention is implemented as follows:

[0006] The first aspect of this invention provides a method for evaluating the overall quality of night vision fusion images, comprising the following steps:

[0007] S10: Preprocess the infrared and visible light color night vision fusion images in the image database to remove thermal targets from the images;

[0008] S20: Divide the preprocessed images into training and testing sets;

[0009] S30: Construct a support vector regressor for image feature extraction and fusion. The support vector regressor includes: an image color feature extraction module, a sharpness feature extraction module, a deep feature extraction module, and a feature fusion module.

[0010] S40: Input the training set into the support vector regressor for training;

[0011] S50: Input the image to be evaluated into the trained support vector regressor to obtain the quality score of the input image.

[0012] Based on the above technical solution, the night vision fusion image comprehensive quality evaluation method of the present invention can be further improved as follows:

[0013] The image color feature extraction module first performs local contrast normalization on the input image, calculates the gray-level co-occurrence matrix corresponding to the R, G, and B channels of the normalized image, and extracts the statistical features of the gray-level co-occurrence matrix as the image color features.

[0014] The sharpness feature extraction module divides the input image into multiple regions of equal size, calculates the equivalent rotation-invariant local binary pattern histogram for each region, and weights the equivalent rotation-invariant local binary pattern histograms as sharpness features.

[0015] The deep feature extraction module uses the MobileNetV3 network to extract deep features from the image.

[0016] The feature fusion module integrates and fuses image color features, image sharpness features, and image deep features.

[0017] In step S10, the preprocessing method for the fused image is to mark the region in the infrared image that exceeds a certain threshold as a hot target, and then replace the pixel value in the hot target region in the corresponding fused image with the average value of eight neighbors.

[0018] Furthermore, in step S30, the image color feature extraction module extracts three statistical features—homogeneity of the gray-level co-occurrence matrix, second-order angular moment, and contrast—as image color features.

[0019] Furthermore, the method for weighting the equivalent rotation-invariant local binary pattern histogram in step S30 is as follows: using the MobileNetV3 network to obtain the gradient class activation map heatmap of the input image, dividing the gradient class activation map heatmap into multiple regions of the same size, calculating the pixel mean of each region of the gradient class activation map heatmap, and weighting the rotation-invariant equivalent binary pattern histogram of the corresponding region in the input image.

[0020] Furthermore, step S30 also includes:

[0021] Step 1: Divide the original fused image and gradient class activation map heatmap into 12 non-overlapping 80×80 image blocks, and calculate the rotation-invariant equivalent local binary pattern histogram for each original fused image block.

[0022] Step 2: Calculate the pixel mean of each gradient class activation map heatmap image patch, and weight the rotation-invariant equivalent binary mode histogram of the corresponding image patch in the original fused image to obtain the final sharpness feature.

[0023] Step 3: Deep Feature Extraction: Extract deep features of the image using the MobileNetV3 network;

[0024] Step 4: Connect and fuse image color features, image sharpness features, and image deep features.

[0025] A second aspect of the present invention provides a computer-readable storage medium, wherein computer program instructions are stored on the computer-readable storage medium; when the computer program instructions are executed by a processor, they implement the above-described method for comprehensive quality evaluation of night vision fusion images.

[0026] A third aspect of the present invention provides a night vision fusion image comprehensive quality evaluation system, wherein the system includes the aforementioned computer-readable storage medium.

[0027] Compared with existing technologies, the beneficial effects of the night vision fusion image comprehensive quality evaluation method, medium, and system provided by this invention are:

[0028] 1. This invention attempts to utilize the correlation between the gray-level co-occurrence matrix and image color to extract the homogeneity, second moment of the angle, and contrast of the gray-level co-occurrence matrix to measure image color quality. Compared with traditional methods, the method of this invention is more sensitive to image color distortion.

[0029] 2. When extracting image features, this invention fully considers the characteristics of human vision, making the evaluation effect of the method more consistent with the subjective perception of the human eye: by performing local contrast normalization processing on the image, the nonlinear masking effect of human vision is simulated; by using class activation heatmap to weight the equivalent rotation-invariant local binary pattern histogram, the selective attention mechanism of human vision is simulated. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating the steps of a night vision fusion image comprehensive quality evaluation method provided by the present invention;

[0032] Figure 2 This invention provides an overall model diagram of a night vision fusion image comprehensive quality evaluation method.

[0033] Figure 3 Image preprocessing results of a night vision fusion image comprehensive quality evaluation method provided by the present invention;

[0034] Figure 4 The image shows a heatmap rendering of a night vision fusion image comprehensive quality evaluation method provided by the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0037] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0038] like Figure 1 The diagram shows the steps of a first embodiment of a night vision fusion image comprehensive quality evaluation method provided by the present invention. Figure 2 This is an overall model diagram of this embodiment. This embodiment includes the following steps:

[0039] S10: Preprocess the infrared and visible light color night vision fusion images in the image database to remove thermal targets from the images;

[0040] S20: Divide the preprocessed images into training and testing sets;

[0041] S30: Construct a support vector regressor for image feature extraction and fusion. The support vector regressor includes: an image color feature extraction module, a sharpness feature extraction module, a deep feature extraction module, and a feature fusion module.

[0042] S40: Input the training set into the support vector regressor for training;

[0043] S50: Input the image to be evaluated into the trained support vector regressor to obtain the quality score of the input image.

[0044] In the above technical solution, the image color feature extraction module first performs local contrast normalization on the input image, calculates the gray-level co-occurrence matrix corresponding to the R, G, and B channels of the normalized image, and extracts the statistical features of the gray-level co-occurrence matrix as the image color features. The image preprocessing result is as follows: Figure 3 As shown.

[0045] The calculation method for local contrast normalization of the input image is as follows:

[0046]

[0047] In the formula, i and j are the spatial indices of the image, and c = 1. When the denominator approaches zero, c is introduced to ensure stability.

[0048]

[0049]

[0050] Where ω={ω k，l |k=-K,...K,l=-L,...L} is a two-dimensional circularly symmetric Gaussian weighted function. In this paper, we set K=L=3.

[0051] In the above technical solution, the sharpness feature extraction module divides the input image into multiple regions of the same size, calculates the equivalent rotation-invariant local binary pattern histogram for each region, and weights the equivalent rotation-invariant local binary pattern histogram as the sharpness feature.

[0052] In the above technical solution, the deep feature extraction module uses the MobileNetV3 network to extract deep features of the image.

[0053] In the above technical solution, the feature fusion module fused image color features, image sharpness features, and image deep features in series.

[0054] In the above technical solution, the method for preprocessing the fused image in step S10 is to mark the region in the infrared image that exceeds a certain threshold as a hot target, and then replace the pixel value in the hot target region in the corresponding fused image with the average value of eight neighbors.

[0055] Furthermore, in the above technical solution, the image color feature extraction module in step S30 extracts three statistical features—homogeneity of the gray-level co-occurrence matrix, second-order angular moment, and contrast—as image color features.

[0056] The image's gray-level co-occurrence matrix homogeneity (ASM), angular second moment (Homogeneity), and contrast are calculated as image color features. The calculations for homogeneity, angular second moment, and contrast are as follows:

[0057]

[0058]

[0059]

[0060]

[0061] In the formula, N is the size of GLCM, and P(l,m) is the probability of GLCM(l,m) occurring in the gray-level co-occurrence matrix.

[0062] Furthermore, in the above technical solution, the method for weighting the equivalent rotation-invariant local binary pattern histogram in step S30 is as follows: the gradient class activation map heatmap of the input image is obtained using the MobileNetV3 network, the gradient class activation map heatmap is divided into multiple regions of the same size, the pixel mean of each region of the gradient class activation map heatmap is calculated, and the rotation-invariant equivalent binary pattern histogram of the corresponding region in the input image is weighted.

[0063] The gradient class activation map heatmap of the original fused image is obtained using the MobileNetV3 network, as shown in the following figure:

[0064]

[0065]

[0066] Where A represents a feature layer, and in this paper A represents the last feature map of the MobileNetV3 network, k represents the kth channel in A, c represents the spatial index of the data in the kth channel of class c, i, j, Z represents the product of the width and height of A, and yc represents the prediction score of MobileNetV3 for class c.

[0067] Heatmap effect as follows Figure 4 As shown.

[0068] Step S30 further includes:

[0069] Step 1: Divide the original fused image and gradient class activation map heatmap into 12 non-overlapping 80×80 image blocks. Calculate the rotation-invariant equivalent local binary pattern histogram for each original fused image block, as follows:

[0070]

[0071] Among them, g c It is the grayscale value of the center pixel, g i U(LBP) represents the grayscale value of adjacent pixels, p is the number of adjacent pixels to be calculated, and U(LBP) represents the number of adjacent pixels to be calculated. c，p ) is the number of transitions from 0 to 1 or from 1 to 0 in the cyclic binary number corresponding to LBP.

[0072] Step 2: Calculate the pixel mean of each gradient class activation map heatmap image patch, and weight the rotation-invariant equivalent binary mode histogram of the corresponding image patch in the original fused image to obtain the final sharpness feature.

[0073] Step 3: Deep Feature Extraction: Extract deep features of the image using the MobileNetV3 network;

[0074] Step 4: Connect and fuse image color features, image sharpness features, and image deep features.

[0075] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for evaluating the overall quality of night vision fusion images, characterized in that, Includes the following steps: S10: Preprocess the infrared and visible light color night vision fusion images in the image database to remove thermal targets from the images; S20: Divide the preprocessed images into training and testing sets; S30: Construct a support vector regressor for image feature extraction and fusion. The support vector regressor includes: an image color feature extraction module, a sharpness feature extraction module, a deep feature extraction module, and a feature fusion module. S40: Input the training set into the support vector regressor for training; S50: Input the image to be evaluated into the trained support vector regressor to obtain the quality score of the input image; The image color feature extraction module is used to extract three statistical features of the gray-level co-occurrence matrix as image color features: homogeneity, second moment of angle, and contrast. The image color feature extraction module first performs local contrast normalization on the input image, calculates the gray-level co-occurrence matrix corresponding to the R, G, and B channels of the normalized image, and extracts the statistical features of the gray-level co-occurrence matrix as image color features. The sharpness feature extraction module divides the input image into multiple regions of equal size, calculates the equivalent rotation-invariant local binary pattern histogram for each region, and weights the equivalent rotation-invariant local binary pattern histograms as sharpness features. The weighting of the equivalent rotation-invariant local binary pattern histograms involves using the MobileNetV3 network to obtain a gradient class activation map heatmap of the input image, dividing the gradient class activation map heatmap into multiple regions of equal size, calculating the pixel mean of each region in the gradient class activation map heatmap, and weighting the rotation-invariant equivalent binary pattern histograms of the corresponding regions in the input image. The homogeneity, second moment (ASM), and contrast of the image's gray-level co-occurrence matrix are calculated as image color features. The calculations for homogeneity, ASM, and contrast are as follows: ; ; ; ; In the formula, N is the size of GLCM, P(l,m) is the probability of GLCM(l,m) occurring in the gray-level co-occurrence matrix, and GLCM is the gray-level co-occurrence matrix; The gradient class activation map heatmap of the original fused image is obtained using the MobileNetV3 network, and is calculated as follows: ; ; Where A represents a feature layer, in this paper A represents the last feature map of the MobileNetV3 network, k represents the k-th channel in A, c represents the class c, i, j are the spatial indices of the data in the k-th channel, and Z represents the product of the width and height of A. This represents the prediction score of MobileNetV3 for category c; The sharpness feature extraction module divides the input image into multiple regions of equal size and calculates the equivalent rotation-invariant local binary pattern histogram for each region. Specifically, it divides the original fused image and gradient class activation map heatmap into 12 non-overlapping 80×80 image blocks and calculates the rotation-invariant equivalent local binary pattern histogram for each image block of the original fused image, as follows: in, It is the grayscale value of the center pixel. is the grayscale value of the adjacent pixel, and p is the number of adjacent pixels to be calculated. It is the number of transitions from 0 to 1 or from 1 to 0 in the cyclic binary number corresponding to LBP.

2. The method for evaluating the overall quality of night vision fusion images according to claim 1, characterized in that, The feature fusion module integrates and fuses image color features, image sharpness features, and image deep features.

3. The method for evaluating the overall quality of night vision fusion images according to claim 1, characterized in that, The method for preprocessing the fused image in step S10 is to mark the region in the infrared image that exceeds a certain threshold as a hot target, and then replace the pixel value in the hot target region in the corresponding fused image with the average value of eight neighbors.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, they implement a night vision fusion image comprehensive quality evaluation method as described in any one of claims 1 to 3.

5. A night vision fusion image comprehensive quality evaluation system, characterized in that, It includes the computer-readable storage medium as described in claim 4.

Images