Object recognition method based on cross-domain fusion of multi-source images

Abstract

The present invention discloses a target recognition method based on cross-domain fusion of multi-source images. The method converts visible light task-related images into pseudo infrared task-related images by calling Pix2pix network and learning the mapping relationship between visible light images and infrared images through the network. In addition, target recognition is performed on pseudo infrared task-related images through a mixed source domain constructed by visible light grayscale / grayscale inversion task-irrelevant images, infrared task-irrelevant images, and visible light grayscale / grayscale inversion task-related images. The method solves the problem of how to improve the classification and recognition accuracy of infrared task-related images in the target domain when infrared data of the task-related class in the target domain cannot be obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of information fusion and target recognition, and in particular relates to a target recognition method based on cross-domain fusion of multi-source images. Background Art

[0002] In information fusion target recognition, infrared imaging sensors and visible light imaging sensors are two common information sources. Since visible light images can reflect feature contours and texture information, and infrared images can reflect the radiation information of objects, the two are complementary to a certain extent, so they are often fused to improve the accuracy of target recognition. However, due to the cost of infrared equipment and possible occlusion, interference and other problems, some data categories are missing in the information source, that is, the categories are incomplete, that is, it is impossible to obtain infrared images of some targets.

[0003] Transfer learning is an important method to solve the problem of insufficient training data in machine learning. It transfers knowledge from the source domain to the target domain. The source domain (source domain) refers to the data with known labels, which is the object to be transferred, while the target domain (target domain) is the object to be endowed with knowledge and annotations. In image target recognition, it is generally believed that the information about the target exists in different domains captured by different imaging sensors.

[0004] When the target recognition method in a certain source domain is applied to other domains (for example, the method of classifying and identifying visible light images is applied to infrared images), the performance of the solution learned from the source domain often decreases, which is caused by the domain shift in typical domain adaptation tasks. Existing methods for cross-domain recognition include zero-sample deep domain adaptation methods and domain generalization methods. The zero-sample deep domain adaptation method divides the data into task-related data (targets to be identified) and task-irrelevant data (non-identified targets), and uses domain adaptation and sensor fusion methods to learn the task-irrelevant source domain and target domain dual-domain training data, and classifies and identifies the task-related data in the target domain. The domain generalization method does not provide or use task-irrelevant data, but learns a model with strong generalization ability through the source domain data and tests it on the target domain.

[0005] It should be noted that both the zero-sample deep domain adaptation method and the domain generalization method have the technical problem of how to improve the accuracy of target classification and recognition in infrared images when infrared data of task-related classes in the target domain cannot be obtained. Summary of the invention

[0006] The purpose of the present invention is to provide a target recognition method based on cross-domain fusion of multi-source images to solve the problem that the classification and recognition of infrared task-related images in the target domain cannot be realized when the infrared data of the task-related classes in the target domain cannot be obtained.

[0007] The present invention adopts the following technical solutions:

[0008] The present invention provides a target recognition method based on cross-domain fusion of multi-source images, the method comprising the following steps:

[0009] Step 1: Call the Pix2pix network to convert the visible light task-irrelevant images into pseudo-infrared task-irrelevant images; perform grayscale and grayscale inversion conversion on the visible light task-irrelevant images to obtain grayscale and grayscale inversion task-irrelevant images; combine the grayscale or grayscale inversion task-irrelevant images and the pseudo-infrared task-irrelevant images to form source domain II;

[0010] Step 2: Perform task-irrelevant image similarity evaluation on the grayscale and grayscale inverted task-irrelevant images and the infrared task-irrelevant images, and obtain the task-irrelevant image similarity evaluation results. The task-irrelevant image similarity evaluation results are used to characterize the similarity between the grayscale image or grayscale inverted image and the infrared image respectively.

[0011] According to the similarity evaluation results of the task-irrelevant images and the visible light task-related images, grayscale or grayscale inversion task-related images are obtained; at the same time, the Pix2pix network is called to convert the visible light task-related images into pseudo infrared task-related images; the grayscale or grayscale inversion task-related images and the pseudo infrared task-related images are combined to form the source domain I;

[0012] Step 3: Use source domain I, source domain II, and source domain III as inputs of the cross-domain image target recognition network, and obtain the first data feature of source domain I, the second data feature of source domain II, and the third data feature of source domain III through feature extraction, wherein the source domain III is composed of copying the infrared task-irrelevant image in step 2 to obtain two groups of infrared task-irrelevant images;

[0013] Classify the pseudo infrared task related images according to the first data feature and calculate the classification loss;

[0014] Simultaneously, distribution measurement is performed on the second data feature and the third data feature, and distribution measurement loss is calculated;

[0015] According to the total loss consisting of the distribution metric loss and the classification loss, the first data feature is updated to obtain the fourth data feature of the source domain I;

[0016] The pseudo-infrared task-related images are classified according to the fourth data feature.

[0017] Optionally, the method for performing grayscale and grayscale inversion conversion on visible light task-independent images includes:

[0018] Get the pixel value of each pixel in the three color channels of RGB of the visible light task-independent image;

[0019] Form a grayscale task-independent image according to the pixel value of each pixel;

[0020] By comparing the maximum pixel value with the pixel value of each pixel in each channel, an image irrelevant to the grayscale inversion task is obtained.

[0021] Optionally, the method of performing task-irrelevant image similarity evaluation on the grayscale and grayscale inversion task-irrelevant images and the infrared task-irrelevant images respectively includes:

[0022] The root mean square error, peak signal-to-noise ratio, structural similarity, Frechette starting distance and comprehensive evaluation results of learnable perceptual image block similarity of grayscale task-irrelevant images, grayscale inversion task-irrelevant images and infrared task-irrelevant images were calculated respectively.

[0023] Based on the comprehensive evaluation results, the similarity evaluation results of task-independent images are obtained.

[0024] Optionally, the total loss of the cross-domain image object recognition network in step 3 is the weighted sum of the classification loss and the distribution alignment loss, and the total loss is specifically:

[0025] L=L cla +λL dis ,

[0026] Among them, L cla is the classification loss, L dis is the classification alignment loss, λ is the hyperparameter for balancing the classification loss and distribution alignment loss, and L is the total loss of the cross-domain image object recognition network.

[0027] Optionally, the classification loss is calculated as:

[0028]

[0029] in, represents the i-th sample in source domain I, represents the sample label of the i-th sample in the source domain I, P represents the classification probability of the neural network output sample, represents the number of samples contained in the source domain I, L cla is the classification loss.

[0030] Optionally, the distribution alignment loss is calculated as:

[0031]

[0032] in, represents the jth sample in source domain II, represents the jth sample in source domain III, Represents the number of samples contained in source domain II, represents the number of samples contained in the source domain III, f represents the feature output of the sample after passing through the feature extractor, represents the regenerated Hilbert space, φ represents the Gaussian kernel function, L dis Represents the image distribution alignment loss of source domain II and source domain III.

[0033] The beneficial effects of the present invention are as follows: first, by calling the Pix2pix network, the mapping relationship between visible light images and infrared images is learned through the network, and the visible light task-related class images are converted into pseudo infrared task-related class images, so that the infrared task-related class images that cannot be obtained are replaced by the pseudo infrared task-related class images; in addition, the grayscale / grayscale inversion task-irrelevant class images, pseudo infrared task-irrelevant class images, infrared task-irrelevant class images, grayscale / grayscale inversion task-related class images obtained by converting visible light task-irrelevant class images, and the obtained pseudo infrared task-related class images are respectively divided into source domains I, source domains II and source domains III, and target recognition is performed on the generated pseudo infrared task-related class images, and auxiliary information is provided in combination with the grayscale and grayscale inversion task lightless class images, so as to improve the target recognition accuracy in the infrared task-related class images. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a generation process of a hybrid source domain provided by the present invention;

[0035] Figure 2 A schematic diagram of a cross-domain fusion target recognition process provided by the present invention; DETAILED DESCRIPTION

[0036] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] It should be noted that the data categories involved in the present invention can be divided into two parts, namely, task-related categories and task-irrelevant categories, and each part obtains a visible light image and an infrared image of the target at the same viewing angle. and Represent the task-related class samples obtained from the source domain and the target domain respectively. Similarly, and They represent task-independent samples obtained from the source domain and the target domain respectively. The main tasks of this invention are: first, it is impossible to obtain infrared task-related images in the target domain in reality, and the Pix2pix network is used to generate pseudo infrared task-related images; second, with the help of labeled information Three parts of sample data for infrared target domain task related data Perform classification and identification.

[0038] In order to obtain pseudo infrared task-related images that are close to infrared task-related images, an image translation network is first constructed. The classic Pix2pix network is used to learn the mapping relationship between visible light and infrared images on task-independent classes, and this mapping relationship is used to convert visible light task-related images into pseudo infrared task-related images.

[0039] The present invention provides a target recognition method based on cross-domain fusion of multi-source images, the method comprising the following steps:

[0040] Step 1: Call the Pix2pix network to convert the visible light task-irrelevant images into pseudo-infrared task-irrelevant images; perform grayscale and grayscale inversion conversion on the visible light task-irrelevant images to obtain grayscale and grayscale inversion task-irrelevant images; combine the grayscale or grayscale inversion task-irrelevant images and the pseudo-infrared task-irrelevant images to form source domain II;

[0041] In one embodiment, if Figure 1 In the method, the Pix2pix network is first called, and the visible light task-related images are used as the input of the trained Pix2pix network. According to the mapping relationship between the visible light and infrared images on the task-irrelevant class, the pseudo infrared task-related images are output. Since the image translation conversion method focuses on achieving style transfer, but ignores the pixel-level conversion, the generated image lacks some detailed texture information, that is, the generated pseudo infrared data after conversion lacks detailed texture information, and the grayscale / grayscale inverted image has the characteristics of the infrared image to a certain extent, and can better retain the detailed information of the visible light image. Therefore, the grayscale / grayscale inverted image is introduced to supplement the image detail information. To this end, the grayscale / grayscale inversion of the visible light task-irrelevant images is further performed to obtain the grayscale and grayscale inverted task-irrelevant images. It should be noted that here it is also necessary to perform task-irrelevant image similarity evaluation on the grayscale and grayscale inversion task-irrelevant images and the infrared task-irrelevant images respectively, to determine whether the grayscale task-irrelevant images or the grayscale inversion task-irrelevant images are more similar to the generated pseudo-infrared task-irrelevant images, so as to obtain a combination of grayscale or grayscale inversion task-irrelevant images and pseudo-infrared task-irrelevant images with higher similarity, which constitutes the source domain II.

[0042] Step 2: Perform task-irrelevant image similarity evaluation on the grayscale and grayscale inverted task-irrelevant images and the infrared task-irrelevant images, and obtain task-irrelevant image similarity evaluation results. The task-irrelevant image similarity evaluation results represent the similarity between the grayscale image or the grayscale inverted image and the infrared image respectively.

[0043] According to the similarity evaluation results of the task-irrelevant images and the visible light task-related images, grayscale or grayscale inversion task-related images are obtained; the grayscale or grayscale inversion task-related images and the pseudo-infrared task-related images are combined to form a source domain I;

[0044] In one embodiment, when generating a pseudo infrared task-related image similar to an infrared image, it is necessary to learn whether the similarity between a grayscale task-irrelevant image and an infrared task-irrelevant image is better, or the similarity between a grayscale inversion task-irrelevant image and an infrared task-irrelevant image is better when generating a pseudo infrared task-irrelevant image. This is used as a similarity assessment of task-irrelevant images to decide which image to use to generate the pseudo infrared task-related image, thereby ensuring that the generated pseudo infrared task-related image has a higher similarity with the grayscale or grayscale inversion task-related image and has a smaller image loss. According to the similarity evaluation results of the task-irrelevant images, for example, if the similarity evaluation results of the task-irrelevant images show that the grayscale task-irrelevant images are more similar to the infrared task-irrelevant images, then when the Pix2pix network is called to convert the visible light task-related images into pseudo infrared task-related images, the visible light task-related images are also converted into grayscale task-related images; if the similarity evaluation results of the task-irrelevant images show that the grayscale inversion task-irrelevant images are more similar to the infrared task-irrelevant images, then the Pix2pix network is called again to convert the visible light task-related images into pseudo infrared task-irrelevant images, and the visible light task-related images are also converted into grayscale inversion task-related images. At this time, according to the similarity results, the corresponding grayscale task-related images or grayscale inversion task-related images are generated and combined with the generated pseudo infrared task-related images to form the source domain I.

[0045] Specifically, Figure 1 In the Pix2pix network, pairs of visible light task-independent images are used. Infrared task-irrelevant images The sample learns the mapping conversion relationship between visible light images (target domain) and infrared images (source domain), and then converts the visible light task-related images into infrared images according to the trained Pix2pix network. The sample is converted to the infrared image target domain to generate a pseudo infrared task-related image (pseudo target domain), denoted as

[0046] Although the conversion using image translation is different from the ideal situation, and here the information between task-irrelevant class data is learned and converted on task-related class samples, the converted image cannot be identified as a perfect infrared image, but it can still learn some useful information that is beneficial to subsequent classification and recognition tasks.

[0047] Step 3: Use source domain I, source domain II, and source domain III as inputs of the cross-domain image target recognition network, and obtain the first data feature of source domain I, the second data feature of source domain II, and the third data feature of source domain III through feature extraction, wherein the source domain III is composed of copying the infrared task-irrelevant image in step 2 to obtain two groups of infrared task-irrelevant images;

[0048] Classify the pseudo infrared task related images according to the first data feature and calculate the classification loss;

[0049] Simultaneously, distribution measurement is performed on the second data feature and the third data feature, and distribution measurement loss is calculated;

[0050] According to the total loss consisting of the distribution metric loss and the classification loss, the first data feature is updated to obtain the fourth data feature of the source domain I;

[0051] The pseudo-infrared task-related images are classified according to the fourth data feature.

[0052] In one embodiment, it should be noted that the function of generating source domain III is similar to that of step 1. Although the task-independent images and the task-dependent images are two different data, they still retain a lot of relevant information for the recognition of task-dependent images. In order to further supplement the association information between visible light images and infrared images, the same number of images as source domain II are supplemented, so that the number of source domains III is consistent with that of source domains II. Figure 1 As shown, for this purpose, the existing infrared task-irrelevant images are copied to form the source domain III.

[0053] It should be noted that in order to better classify and identify the pseudo-infrared task-related images generated in source domain I, source domains II and III are constructed to provide auxiliary information.

[0054] The hybrid domain source domain I, source domain II, and source domain III constructed in the above steps are used as the input of the cross-domain image target recognition network, and the deep domain generalization network is used to comprehensively utilize the information on each source domain for cross-domain fusion target recognition, such as Figure 2As shown, feature extraction is performed on the data stream of source domain I, the data stream of source domain II and the data stream of source domain III respectively, and the first data feature of source domain I, the second data feature of source domain II and the third data feature of source domain III are extracted; then the data of source domain I is classified according to the first data feature of source domain I through a classifier to obtain the initial classification result and calculate the classification loss of source domain I; at the same time, the second data feature of source domain II and the third data feature of source domain III are distributed aligned to obtain the distribution alignment loss of source domain II and source domain III in the distribution measurement result; according to the total loss of the cross-domain image target recognition network composed of the classification loss and the distribution alignment loss, the network parameters of the backbone network are optimized according to a certain gradient, and the feature extraction is performed on the data stream of source domain I again to obtain the fourth data feature of source domain I. The fourth data feature is the data feature after updating the first data feature, and the data of source domain I is reclassified until the classification accuracy requirement is met.

[0055] In this embodiment, the data features of source domain I are mainly used, and the data features of source domain II and source domain III are supplemented, so as to capture the domain-invariant features between the source domain and the target domain with the help of task-independent data of the source domain and the target domain, so as to perform cross-domain recognition tasks, and solve the technical problem that when the infrared data of the task-related class in the target domain cannot be obtained, it is impossible to implement the visible light data of the task-independent class in the source domain to classify and identify the target in the target domain.

[0056] Optionally, the method for performing grayscale and grayscale inversion conversion on visible light task-independent images includes:

[0057] Get the pixel value of each pixel in the three color channels of RGB of the visible light task-independent image;

[0058] Form a grayscale task-independent image according to the pixel value of each pixel;

[0059] By comparing the maximum pixel value with the pixel value of each pixel in each channel, an image irrelevant to the grayscale inversion task is obtained.

[0060] In one embodiment, converting a visible light image into a grayscale / grayscale inverted image is essentially a conversion between corresponding pixel values. For example, each coordinate point (P x ,P y ) is recorded as I(P x ,P y ), and the pixel values ​​of the three color channels of RGB are recorded as I R (P x ,P y ), I G (P x ,P y ) and IB (P x ,P y ), then the pixel value after conversion is I'(P x ,P y ) can be calculated as follows:

[0061]

[0062] Among them, N represents the maximum value of the pixel, which is selected as 255 here, and W R Represents the R channel pixel conversion coefficient, W G Represents the G channel pixel conversion coefficient and W B Represents the B channel pixel conversion coefficient. For example, usually W R Select 0.299, W G Select 0.587 and W G Select 0.114.

[0063] Through the above-mentioned grayscale and grayscale inversion method of visible light images, when the processed object is a visible light task-independent image, the pixel value of each pixel under the three RGB color channels of the visible light task-independent image can be obtained, and the visible light task-independent image can be converted into a grayscale task-independent image, or converted into a grayscale inversion task-independent image.

[0064] It should be noted that the method for converting visible light task-related images into grayscale task-related images or grayscale inversion task-related images is consistent with the above method.

[0065] Optionally, the method of performing task-irrelevant image similarity evaluation on the grayscale and grayscale inversion task-irrelevant images and the infrared task-irrelevant images respectively includes:

[0066] The root mean square error, peak signal-to-noise ratio, structural similarity, Frechette starting distance and comprehensive evaluation results of learnable perceptual image block similarity of grayscale task-irrelevant images, grayscale inversion task-irrelevant images and infrared task-irrelevant images were calculated respectively.

[0067] Based on the comprehensive evaluation results, the similarity evaluation results of task-independent images are obtained.

[0068] In one embodiment, converting a visible light task-related image into a grayscale task-related image or a grayscale inversion task-related image depends on the image similarity evaluation results between the grayscale and grayscale inversion task-related images and the infrared task-irrelevant images respectively during the process of generating a pseudo infrared task-irrelevant image based on the visible light task-irrelevant image.

[0069] Because the task-related image type selection is guided by the similarity evaluation of task-independent images, it is necessary to consider a comprehensive similarity evaluation. Common evaluation indicators that can be used for image type conversion are divided into objective evaluation indicators and subjective evaluation indicators. In terms of objective evaluation indicators, the similarity evaluation indicators selected are root mean square error (RMSE), peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). In addition to objective evaluation indicators, indicators that can reflect subjective visual perception errors are also considered. The evaluation indicators used include Frechet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS). According to these five evaluation criteria, it is statistically found that the grayscale or grayscale inverted images among the five evaluation criteria have a greater similarity with the images unrelated to the infrared task. This is used to determine that when converting visible light task-related images into pseudo-infrared task images, grayscale or grayscale inverted images should be used to make them more similar to the obtained pseudo-infrared task-related images.

[0070] To this end, the specific method for calculating the root mean square error, peak signal-to-noise ratio, structural similarity, Frechette starting distance and the comprehensive evaluation result value of the learnable perceptual image block similarity between the grayscale task-irrelevant image, the grayscale inversion task-irrelevant image and the infrared task-irrelevant image is as follows:

[0071] 1. RMSE can reflect the degree of difference between variables. It is an objective evaluation index of image quality based on pixel error. It is used to measure the difference between the image to be evaluated and the real image. Its calculation expression is as follows:

[0072]

[0073] Where i is the pixel index, N p is the total number of pixels in a single image, y i represents the gray value of pixel i in the infrared task-irrelevant image, They represent the grayscale value of the grayscale or grayscale inversion task-irrelevant image at pixel i respectively.

[0074] 2. PSNR is one of the most widely used evaluation indicators for measuring image reconstruction quality. This indicator is defined by the maximum pixel value and mean square error of the image. Its calculation expression is as follows:

[0075]

[0076] Among them, variables i, N p ,y i and The definition of is the same as that of RMSE, except that MAX I Represents the maximum grayscale value in an image I, which is selected as 255 here.

[0077] 3. SSIM is a method to measure the similarity between the image to be evaluated and the real image. It is also a perception-based model that needs to comprehensively consider the effects of brightness, contrast and structure. Its calculation expression is as follows:

[0078]

[0079] Among them, μ y is the mean and value of the infrared task-irrelevant class image y is the grayscale or grayscale inversion task-irrelevant class image y * The mean value of is the variance and is the grayscale or grayscale inversion task-irrelevant class image y * The variance of is the covariance of the two, c1 is the first non-zero constant and c2 is the second non-zero constant.

[0080] 4. FID evaluation index: The infrared task-irrelevant images and the grayscale or grayscale inversion task-irrelevant images are respectively input into the IncepionV3 image classification model pre-trained on the ImageNet dataset (the last pooling layer is used as the output layer), and the multi-dimensional (here 2048 dimensions) feature vectors f of the infrared task-irrelevant images and the grayscale or grayscale inversion task-irrelevant images are obtained. real and f fake After that, the error calculation is performed, and the FID calculation expression is as follows:

[0081]

[0082] Among them, f real is the feature vector of the infrared task-irrelevant image, f fake is the feature vector of the image irrelevant to the grayscale or grayscale inversion task, and m1 is the feature vector f real The characteristic mean of m2 is the characteristic vector f fake The characteristic mean of C1 is the characteristic vector f real The covariance matrix of fake The covariance matrix of , Tr is the trace of the matrix.

[0083] 5. The LPIPS evaluation index forces the generator to learn the reverse mapping of the infrared task-independent image reconstructed from the grayscale or grayscale inverted task-independent image, and prioritizes the perceptual similarity between the two. It extracts feature stacks from the l-layer network and normalizes them to units in the channel dimension, using the vector ω l The activation channels are scaled and the L2 distance is calculated. Finally, the features are averaged in space and summed over the channels. The calculation expression is as follows:

[0084]

[0085] Among them, l represents the number of network layers, The feature map and is the feature map extracted from the grayscale / grayscale inversion task-irrelevant image after passing through the l-layer network, ω l The feature map and The weight in each channel, h represents the height of the feature map and ω represents the width of the feature map.

[0086] According to the above five similarity evaluation indicators, the grayscale and grayscale inverted images of the task-irrelevant class are evaluated for similarity with the infrared images of the task-irrelevant class, the conversion method most similar to the infrared image is selected, and the comprehensive evaluation results corresponding to the grayscale task-irrelevant class images and the grayscale inverted task-irrelevant class images are statistically analyzed. For example, if four of the five indicators show that the grayscale task-irrelevant class images are similar to the infrared task-irrelevant class images, then for the process of generating pseudo-infrared task-related class images from task-related class images, it is chosen to convert the visible light task-related class images into grayscale task-related class images; otherwise, for the process of generating pseudo-infrared task-related class images from task-related class images, it is chosen to convert the visible light task-related class images into grayscale inverted task-related class images.

[0087] Optionally, the classification loss in step 3 is calculated as follows:

[0088] In one embodiment, a cross-domain image object recognition network is designed by comprehensively utilizing information between source domains. The design idea of ​​the loss function will be specifically introduced below.

[0089] For the source domain I for classification and recognition, the empirical risk minimization strategy is used when designing the classification loss. The purpose of this strategy is to input the real infrared image into the network and perform classification test when the minimum classification loss is achieved on the source domain I during the sample training phase. cla It is calculated as follows:

[0090]

[0091] in, represents the i-th sample in source domain I, represents the sample label of the i-th sample in the source domain I, P represents the classification probability of the neural network output sample, represents the number of samples contained in the source domain I, L cla is the classification loss.

[0092] Optionally, the distribution alignment loss in step 3 is calculated as follows:

[0093] For source domains II and III for auxiliary recognition, in order to capture as many common features between the two domains as possible, distribution alignment is achieved. Although the sample categories in source domains II and III are both task-independent, the domain-related information in the two domains is useful for cross-domain recognition. A distribution alignment loss is designed in the network, which is measured using the maximum mean difference distance. The distribution metric loss L is designed to dis The calculation formula is as follows:

[0094]

[0095] in, represents the jth sample in source domain II, represents the jth sample in source domain III, Represents the number of samples contained in source domain II, represents the number of samples contained in the source domain III, f represents the feature output of the sample after passing through the feature extractor, represents the regenerated Hilbert space, φ represents the Gaussian kernel function, L dis Represents the image distribution alignment loss of source domain II and source domain III.

[0096] It should be noted that the loss L of the proposed cross-domain image object recognition network finally consists of two parts of loss, namely, the classification loss L of the task-related class samples in the source domain I and cla The distribution metric loss L is calculated between the task-irrelevant samples in source domain II and source domain III. dis , and its calculation formula is as follows:

[0097] L=L cla +λL dis (9)

[0098] Among them, λ is a hyperparameter that balances the above classification loss and distribution alignment loss.

[0099] The cross-domain image recognition network can be trained using source domain data and identify pseudo-infrared task-related image data. Its loss function takes into account the classification loss of source domain I and the distribution measurement loss between source domains II and III. The classification loss follows the empirical risk minimization criterion, and the distribution alignment loss is to capture the common features between the mixed source domains (pseudo-infrared task-irrelevant images and grayscale / grayscale inversion task-irrelevant images) and the real infrared task-irrelevant images, thereby improving the target recognition accuracy.

[0100] The present invention also provides a training method for a Pix2pix network, so that a visible light image can be input and a pseudo infrared image can be output.

[0101] In the training phase, images irrelevant to the infrared task, images irrelevant to the visible light task, and images relevant to the visible light task can be obtained. First, the mapping relationship between the task irrelevant images in the visible light and infrared image domains can be learned. If the visible light task relevant images are converted into pseudo infrared task relevant images according to the learned mapping relationship, it will be beneficial to the subsequent classification and recognition tasks of the infrared task relevant classes.

[0102] The most essential task of the above image conversion is to predict pixels from pixels, which belongs to the category of image translation. The most classic image translation method is the Pix2pix network, which is the original general framework for pixel-level conversion and can achieve style transfer between different images. The scene in the Pix2pix network is essentially similar or even easier, because visible light images often contain more texture details and scene information than infrared images.

[0103] It should be noted that the Pix2pix network consists of two parts: the generator and the discriminator. The generator is used to generate images, while the discriminator is used to identify the authenticity of the images generated by the generator. * It consists of two parts, one of which is conditional adversarial loss:

[0104] L cGAN (G,D)=E x,y [logD(x,y)]+E x,z [log(1-D(x,G(x,z)))] (10)

[0105] Among them, G represents the generator, D represents the discriminator, x represents the input data, y represents the output data, and z represents Gaussian noise.

[0106] It should be noted that in the training phase, the input data here is selected as images irrelevant to the visible light task. The output data here is a pseudo infrared task-irrelevant image

[0107] The target loss L of the Pix2pix network * The other part is the L1 pixel loss:

[0108] L L1 (G) = E x,y,z [||yG(x,z)||1] (11)

[0109] Among them, ||yG(x,z)||1 represents the L1 norm between the output of the generator and the input visible light task-independent class image.

[0110] The final target loss L of the Pix2pix network * as follows:

[0111]

[0112] Among them, the generator G tries to reduce the target loss while the discriminator D tries to increase it, and α is a balancing coefficient used to balance the conditional adversarial loss and pixel loss.

[0113] The present invention also provides a feasibility verification experiment of target recognition based on cross-domain fusion of multi-source images, the specific contents of which are as follows:

[0114] For the cross-domain recognition task between visible light images and infrared images, the VAIS ship dataset containing paired visible light and infrared images was selected for experiments, as shown in Table 1. In order to further simulate the real recognition task, the following experimental scenarios were set in the VAIS ship dataset during the experimental setting:

[0115] Scenario ①: Select images of three categories with a small number of samples (medium-sized other ships, passenger ships, and tugboats) as task-irrelevant images, and identify the remaining three categories with a large number of samples (cargo ships, sailboats, and small ships) as task-related images.

[0116] Table 1 Basic information of VAIS ship dataset

[0117]

[0118]

[0119] Evaluation indicators

[0120] Taking into account the imbalance of categories in the VAIS ship dataset, in order to comprehensively evaluate the performance of this target recognition method, the experiment will use four evaluation indicators: precision, recall, F1 score, and accuracy to evaluate the model, among which the F1 score and accuracy are mainly targeted. Precision and recall mainly indicate that the F1 score can be obtained after calculation.

[0121] The definition of each indicator is now given. In the definition, TP represents the number of true positive examples (TruePositive) in multi-classification test samples, TN represents the number of true negative examples (TrueNegative) in multi-classification test samples, FP represents the number of false positive examples (FalsePositive) in multi-classification test samples, and FN represents the number of false negative examples (FalseNegative) in multi-classification test samples.

[0122] Among them, precision: represents the proportion of true positive examples predicted as positive examples, and its calculation formula is as follows:

[0123]

[0124] Recall: It indicates the proportion of all positive examples that are correctly classified. The calculation formula is as follows:

[0125]

[0126] F1 score (F1Score): represents the harmonic mean of precision and recall, and its calculation formula is as follows:

[0127]

[0128] Accuracy: It indicates the proportion of correctly classified samples to the total samples. The calculation formula is as follows:

[0129]

[0130] Experimental Results

[0131] Table 2 below shows the results of similarity evaluation of the VAIS ship dataset for task-independent samples and infrared task-independent images. The ↑ symbol is placed after the indicator with a larger value, the better the similarity, and vice versa.

[0132] Table 2 Similarity evaluation results of task-independent images in the VAIS ship dataset

[0133]

[0134]

[0135] From Table 2, we can find that in the VAIS ship dataset, except for the FID index, the subjective and objective similarity indexes of other grayscale inversion images are superior to those of grayscale images. Therefore, when performing source domain mixing based on image type conversion in this dataset scenario, the conversion method of converting visible light task-related images into grayscale inversion task-related images is adopted.

[0136] Representative methods are selected from two aspects: the solution to the deep domain generalization problem and the solution to the zero-shot deep domain adaptation problem for comparative experiments. Among the deep domain generalization methods, four representative methods, including Empirical Risk Minimization (ERM), logical AND Mask (ANDMask), Representation Self-Challenging (RSC) and Mixup learning principle, are selected for comparative experiments with this method. Among the zero-shot deep domain adaptation methods, a zero-shot domain adaptation method based on decoupled feature collaborative learning is selected for experiments.

[0137] A comparative experiment was conducted on the VAIS ship dataset. 80% of the data in the source domain was selected as the training set, 20% as the validation set, and the task-related data in the target domain was used as the test set to calculate the average and standard deviation of various indicators. The experimental results are shown in Table 3. Among them, DF-ZSDA, which is used as a comparative method, can only achieve zero-sample domain adaptation from a single source domain to the target domain. At the same time, its task-irrelevant class must be the majority class (the number of task-irrelevant classes>2), so the comparative experiment cannot be performed in the partial recognition case of scene ①.

[0138] Table 3 Experimental results of classification and recognition of VAIS ship dataset under scenario ① (%)

[0139]

[0140]

[0141]

[0142] It can be found from Table 3 that in most task scenarios, the proposed target recognition methods based on cross-domain fusion of multi-source images can achieve experimental performance superior to other methods. When the four deep domain generalization methods (ERM, ANDMask, RSC and Mixup) are trained on a single visible light image or a single grayscale / grayscale inverted image domain, it is difficult to learn a model with strong generalization ability due to the small number of source domains. At the same time, there is a certain domain offset between the source domain and the target domain. If training is performed on the source domain image, it is inevitable that the classification and recognition effect of the target domain image will be poor. From the perspective of information utilization, the deep domain generalization method does not utilize the information of the source domain task-irrelevant class images and the target domain task-irrelevant class images, and cannot make full use of the existing information. In the VAIS ship dataset, the deep domain generalization results using multiple source domain images are not much different from those of the deep domain generalization using a single source domain. This is because multiple source domains (visible light images, grayscale images) do not form a good complementary information complement.

[0143] For the zero-sample deep domain adaptation method DF-ZSDA, its usage scenario is limited to a single source domain and the task-irrelevant class must be the majority class. In the VAIS ship dataset, its recognition performance is better than other deep domain generalization methods under the same settings. Generally, if it is possible to achieve good classification and recognition performance on source domain task-related images, source domain task-irrelevant images, and target domain task-irrelevant images at the same time, then such a method has a certain degree of generalization and may also lose some classification characteristics for target domain task-related images, that is, this method may cause the problem of "broad but not precise" in classification.

[0144] This method fully exploits the information correlation between the source domain and the target domain. Whether it is image translation or grayscale / grayscale inversion operation, it can make full use of the information of existing source domain task-irrelevant images, source domain task-related images, and target domain task-irrelevant images. At the same time, the source domain mixing based on image type conversion can maximize the use of existing information. The designed deep domain generalization target recognition method takes into account the risk minimization strategy and the ability to extract common features between the source domain and the target domain, which can better complete the task of domain generalization and identification of target domain task-related classes. In the VAIS ship dataset, the ResNet18 backbone network is used in the scene ① classification and recognition task. Compared with other comparison methods, at least 6.72% F1 score improvement and 5.65% accuracy improvement are achieved; using the ResNet50 backbone network, at least 12.94% F1 score improvement and 11.41% accuracy improvement are achieved; at the same time, it can be found that this method has a higher F1 score and accuracy, and its corresponding standard deviation value is also small, indicating that this method has a good target recognition accuracy while also having a certain model robustness.

Claims

1. A target recognition method based on cross-domain fusion of multi-source images, characterized in that: The method comprises the following steps: Step 1: Call the Pix2pix network to convert the visible light task-irrelevant image into a pseudo infrared task-irrelevant image; Performing grayscale and grayscale inversion conversion on the visible light task-irrelevant image to obtain grayscale and grayscale inversion task-irrelevant images; combining the grayscale or grayscale inversion task-irrelevant image and the pseudo-infrared task-irrelevant image to form a source domain II; Step 2: performing task-independent image similarity evaluation on the grayscale and grayscale inverted task-independent images and the infrared task-independent images, respectively, to obtain task-independent image similarity evaluation results, wherein the task-independent image similarity evaluation results are used to characterize the similarity between the grayscale image or the grayscale inverted image and the infrared image, respectively; According to the similarity evaluation results of the task-independent images and the visible light task-related images, grayscale or grayscale inversion task-related images are obtained; at the same time, the Pix2pix network is called to convert the visible light task-related images into pseudo infrared task-related images; the grayscale or grayscale inversion task-related images and the pseudo infrared task-related images are combined to form a source domain I; Copy the infrared task-irrelevant images in step 2 to obtain two groups of infrared task-irrelevant images to form source domain III; Step 3: Using the source domain I, the source domain II, and the source domain III as inputs of the cross-domain image target recognition network, respectively, and obtaining the first data feature of the source domain I, the second data feature of the source domain II, and the third data feature of the source domain III through feature extraction, wherein the source domain III is composed of copying the infrared task-irrelevant image in step 2 to obtain two groups of infrared task-irrelevant images; Classify the pseudo infrared task related images according to the first data feature, and calculate the classification loss; Simultaneously performing distribution measurement on the second data feature and the third data feature, and calculating distribution measurement loss; According to the total loss consisting of the distribution metric loss and the classification loss, updating the first data feature to obtain a fourth data feature of the source domain I; The pseudo-infrared task-related image is classified according to the fourth data feature.

2. The target recognition method based on multi-source image cross-domain fusion as claimed in claim 1, characterized in that: Methods for grayscale and grayscale inversion conversion of visible light task-independent images include: Obtain the pixel value of each pixel in the three color channels of RGB of the visible light task-independent image; Forming the grayscale task-independent image according to the pixel value of each pixel; By comparing the maximum pixel value with the pixel value of each pixel in each channel, an image irrelevant to the grayscale inversion task is obtained.

3. The target recognition method based on multi-source image cross-domain fusion according to claim 1, characterized in that: The method for performing task-irrelevant image similarity evaluation on the grayscale and grayscale inversion task-irrelevant images and the infrared task-irrelevant images respectively includes: Calculate the root mean square error, peak signal-to-noise ratio, structural similarity, Frechette starting distance and comprehensive evaluation results of learnable perceptual image block similarity of the grayscale task-irrelevant image, the grayscale inversion task-irrelevant image and the infrared task-irrelevant image respectively; The task-independent image similarity evaluation result is obtained according to the comprehensive evaluation result.

4. The target recognition method based on multi-source image cross-domain fusion according to claim 1, characterized in that: The total loss of the cross-domain image object recognition network in step 3 is the weighted sum of the classification loss and the distribution alignment loss. The total loss is specifically: L=L cla +λL dis , Among them, L cla is the classification loss, L dis is the classification alignment loss, λ is a hyperparameter that balances the classification loss and the distribution alignment loss, and L is the total loss of the cross-domain image target recognition network.

5. The target recognition method based on multi-source image cross-domain fusion as claimed in claim 4, characterized in that: The classification loss is calculated as: in, represents the i-th sample in source domain I, represents the sample label of the i-th sample in the source domain I, P represents the classification probability of the neural network output sample, represents the number of samples contained in the source domain I, L cla is the classification loss.

6. The target recognition method based on multi-source image cross-domain fusion as claimed in claim 4, characterized in that: The distribution alignment loss is calculated as: in, represents the jth sample in source domain II, represents the jth sample in source domain III, Represents the number of samples contained in source domain II, represents the number of samples contained in the source domain III, f represents the feature output of the sample after passing through the feature extractor, represents the regenerated Hilbert space, φ represents the Gaussian kernel function, L dis Represents the image distribution alignment loss of source domain II and source domain III.

Images