Training method of image processing model, image processing method, device and medium

Abstract

The present disclosure provides a training method of an image processing model, an image processing method, an apparatus and a medium. The training method comprises: obtaining a plurality of groups of preset sample images; each group of preset sample images comprises a first sample infrared image, a second sample infrared image and a sample visible light image under the same sample view angle, the resolution of the first sample infrared image is lower than the resolution of the second sample infrared image and the resolution of the sample visible light image; and a preset network model is trained based on each group of preset sample images to obtain an image processing model. The present disclosure can combine a low-resolution infrared image with a visible light image to obtain a high-resolution infrared image, thereby saving costs while expanding the information of the infrared image; and through multiple levels of convolution, pooling and up-sampling processing, the details and overall features of the image are taken into account, the accuracy of the output high-resolution infrared image is ensured, and the improvement of the resolution of the infrared image is accurately and efficiently realized.

Description

Technical Field

[0001] This disclosure belongs to the field of image processing technology, and in particular relates to a training method for an image processing model, an image processing method, an apparatus, and a medium. Background Technology

[0002] Infrared sensors can detect the temperature distribution of a target area and output infrared images. They are widely used in many fields such as non-contact temperature measurement, gas composition analysis, and non-destructive testing. When the infrared temperature field to be detected is relatively fine, a high-resolution infrared sensor is required. However, the cost of both thermopile array sensors and microthermometer sensors is directly proportional to the sensor's resolution. Currently, to reduce the cost of sensor use, the main approach is to improve image resolution by interpolating low-resolution infrared images (bilinear interpolation, nearest neighbor interpolation, etc.). However, this method cannot expand the amount of information in the original image, thus failing to improve the fineness of the infrared image information. Summary of the Invention

[0003] The technical problem to be solved by this disclosure is to overcome the shortcomings of existing technologies that use interpolation to improve the resolution of infrared images, which cannot expand the information content of the original image and result in low detail of the infrared image. This disclosure provides a training method for an image processing model, an image processing method, an apparatus and a medium.

[0004] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0005] This disclosure provides a training method for an image processing model, the training method comprising:

[0006] Acquire several sets of preset sample images;

[0007] Each set of preset sample images includes a first sample infrared image, a second sample infrared image, and a sample visible light image from the same sample viewpoint. The resolution of the first sample infrared image is lower than the resolution of the second sample infrared image and the resolution of the sample visible light image.

[0008] Based on the preset sample images of each group, a preset network model is trained to obtain the image processing model.

[0009] Preferably, the step of training a preset network model based on each set of preset sample images to obtain the image processing model includes:

[0010] The first sample infrared image and the sample visible light image of each group are input into the preset network model, and the simulated infrared image is output.

[0011] The resolution of the simulated infrared image is higher than that of the first sample infrared image;

[0012] The loss value of the simulated infrared image compared to the corresponding second sample infrared image is calculated;

[0013] The model parameters of the preset network model are updated based on the loss value until the loss value meets the preset loss condition to obtain the image processing model.

[0014] Preferably, the step of inputting the first sample infrared image and the sample visible light image of each group into the preset network model and outputting a simulated infrared image includes:

[0015] For each set of preset sample images input to the preset network model, several infrared feature maps at a first resolution are obtained based on the first sample infrared image.

[0016] Based on the sample visible light image, several optical feature maps at a second resolution are obtained;

[0017] Each infrared feature map is stitched together with an optical feature map to obtain several fused feature maps at a third resolution.

[0018] Wherein, the resolution difference between the spliced ​​infrared feature map and the optical feature map is less than the first preset resolution difference;

[0019] The simulated infrared image is obtained based on several of the fused feature maps.

[0020] Preferably, the preset network model includes an infrared feature extraction module, an optical feature extraction module, a stitching module, and an image reconstruction module;

[0021] The infrared feature extraction module is used to acquire the infrared feature map;

[0022] The optical feature extraction module is used to acquire the optical feature map;

[0023] The stitching module is used to stitch together the infrared feature map and the optical feature map;

[0024] The image reconstruction module is used to acquire the simulated infrared image.

[0025] Preferably, the preset network model is a convolutional neural network model;

[0026] The step of obtaining several infrared feature maps at a first resolution based on the first sample infrared image includes:

[0027] The first sample infrared image is alternately subjected to convolution and upsampling processing to obtain several infrared feature maps at the first resolution;

[0028] And / or,

[0029] The step of obtaining several optical feature maps at a second resolution based on the sample visible light image includes:

[0030] The sample visible light image is alternately subjected to convolution and pooling processing to obtain several optical feature maps at the second resolution;

[0031] And / or,

[0032] The step of obtaining the simulated infrared image based on the plurality of fused feature maps includes:

[0033] The fused feature map is alternately subjected to convolution and upsampling processes to obtain the simulated infrared image.

[0034] Preferably, the step of alternately performing convolution and upsampling processing on the fused feature map to obtain the simulated infrared image includes:

[0035] The fused feature map with the lowest resolution is alternately processed by convolution and upsampling, and another fused feature map is stitched together after each upsampling process. The process is repeated in order of increasing resolution until all fused feature maps are traversed to obtain the simulated infrared image.

[0036] In each instance, the resolution difference between the two fused feature maps is less than the second preset resolution difference.

[0037] Preferably, the step of stitching each of the infrared feature maps with one of the optical feature maps further includes:

[0038] At least one of the stitched infrared feature map and optical feature map is cropped and / or expanded to make the resolution of the infrared feature map and the optical feature map equal.

[0039] And / or,

[0040] The step of stitching together another fused feature map after each upsampling process also includes:

[0041] At least one of the two fused feature maps is cropped and / or expanded to make the two fused feature maps have the same resolution.

[0042] This disclosure also provides an image processing method, which is implemented based on the image processing model obtained by the above-described training method, and the image processing method includes:

[0043] Acquire the infrared image and visible light image of the first target from the same target viewpoint;

[0044] The first infrared image of the target and the visible light image of the target are input into the image processing model to output the second infrared image of the target;

[0045] The resolution of the second target infrared image is higher than that of the first target infrared image.

[0046] This disclosure also provides a training system for an image processing model, the training system comprising:

[0047] The sample image acquisition module is used to acquire several sets of preset sample images;

[0048] Each set of preset sample images includes a first sample infrared image, a second sample infrared image, and a sample visible light image from the same sample viewpoint. The resolution of the first sample infrared image is lower than the resolution of the second sample infrared image and the resolution of the sample visible light image.

[0049] The model training module is used to train a preset network model based on the preset sample images of each group to obtain the image processing model.

[0050] Preferably, the model training module includes:

[0051] The simulated image output unit is used to input the first sample infrared image and the sample visible light image of each group into the preset network model and output a simulated infrared image.

[0052] The resolution of the simulated infrared image is higher than that of the first sample infrared image;

[0053] The loss calculation unit is used to calculate the loss value of the simulated infrared image compared to the corresponding second sample infrared image;

[0054] The parameter update unit is used to update the model parameters of the preset network model based on the loss value until the loss value meets the preset loss condition to obtain the image processing model.

[0055] Preferably, the analog image output unit is further configured to:

[0056] For each set of preset sample images input to the preset network model, several infrared feature maps at a first resolution are obtained based on the first sample infrared image.

[0057] Based on the sample visible light image, several optical feature maps at a second resolution are obtained;

[0058] Each infrared feature map is stitched together with an optical feature map to obtain several fused feature maps at a third resolution.

[0059] Wherein, the resolution difference between the spliced ​​infrared feature map and the optical feature map is less than the first preset resolution difference;

[0060] The simulated infrared image is obtained based on several of the fused feature maps.

[0061] Preferably, the preset network model includes an infrared feature extraction module, an optical feature extraction module, a stitching module, and an image reconstruction module;

[0062] The infrared feature extraction module is used to acquire the infrared feature map;

[0063] The optical feature extraction module is used to acquire the optical feature map;

[0064] The stitching module is used to stitch together the infrared feature map and the optical feature map;

[0065] The image reconstruction module is used to acquire the simulated infrared image.

[0066] Preferably, the preset network model is a convolutional neural network model;

[0067] The analog image output unit is also used for:

[0068] The first sample infrared image is alternately subjected to convolution and upsampling processing to obtain several infrared feature maps at the first resolution;

[0069] And / or,

[0070] The sample visible light image is alternately subjected to convolution and pooling processing to obtain several optical feature maps at the second resolution;

[0071] And / or,

[0072] The fused feature map is alternately subjected to convolution and upsampling processes to obtain the simulated infrared image.

[0073] Preferably, the analog image output unit is further configured to:

[0074] The fused feature map with the lowest resolution is alternately processed by convolution and upsampling, and another fused feature map is stitched together after each upsampling process. The process is repeated in order of increasing resolution until all fused feature maps are traversed to obtain the simulated infrared image.

[0075] In each instance, the resolution difference between the two fused feature maps is less than the second preset resolution difference.

[0076] Preferably, the analog image output unit is further configured to:

[0077] At least one of the stitched infrared feature map and optical feature map is cropped and / or expanded to make the resolution of the infrared feature map and the optical feature map equal.

[0078] And / or,

[0079] At least one of the two fused feature maps is cropped and / or expanded to make the two fused feature maps have the same resolution.

[0080] This disclosure also provides an image processing system, which is implemented based on the image processing model obtained from the training system described above, and the image processing system includes:

[0081] The target image acquisition module is used to acquire the infrared image and the visible light image of the first target from the same target viewpoint.

[0082] An image processing module is used to input the first target infrared image and the target visible light image into the image processing model to output a second target infrared image;

[0083] The resolution of the second target infrared image is higher than that of the first target infrared image.

[0084] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and used to run on the processor, wherein the processor executes the computer program to implement the above-described image processing model training method and image processing method.

[0085] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described training method for the image processing model and the image processing method.

[0086] Based on common knowledge in the field, the preferred conditions can be combined arbitrarily to obtain the various preferred embodiments of this disclosure.

[0087] The positive advancements of this disclosure are as follows: by training an image processing model, a high-resolution infrared image can be obtained by combining a low-resolution infrared image acquired at low cost with a visible light image, thus expanding the information of the infrared image while saving costs; and through multiple levels of convolution, pooling, and upsampling processing, both the detailed and overall features of the image are taken into account, ensuring the accuracy of the output high-resolution infrared image, thereby accurately and efficiently improving the resolution of the infrared image. Attached Figure Description

[0088] Figure 1 This is a first flowchart of the training method for the image processing model of Embodiment 1 of this disclosure.

[0089] Figure 2 This is a second flowchart of the training method for the image processing model in Embodiment 1 of this disclosure.

[0090] Figure 3 This is the third flowchart of the training method for the image processing model in Embodiment 1 of this disclosure.

[0091] Figure 4 This is a schematic diagram of the image processing model in this disclosure.

[0092] Figure 5 This is a flowchart of the image processing method of Embodiment 2 of this disclosure.

[0093] Figure 6 This is a schematic diagram of the first module of the training system for the image processing model of Embodiment 3 of this disclosure.

[0094] Figure 7 This is a schematic diagram of the second module of the training system for the image processing model of Embodiment 3 of this disclosure.

[0095] Figure 8 This is a schematic diagram of the image processing system of Embodiment 4 of this disclosure.

[0096] Figure 9 This is a schematic diagram of the structure of the electronic device according to Embodiment 5 of this disclosure. Detailed Implementation

[0097] The present disclosure is further illustrated below by way of embodiments, but is not intended to limit the scope of the embodiments.

[0098] Example 1

[0099] This embodiment provides a training method for an image processing model, such as... Figure 1 As shown, the training method includes the following steps:

[0100] S1. Obtain several sets of preset sample images;

[0101] Each set of preset sample images includes a first sample infrared image, a second sample infrared image, and a sample visible light image from the same sample viewpoint. The resolution of the first sample infrared image is lower than that of the second sample infrared image and the sample visible light image.

[0102] S2. Based on each set of preset sample images, train a preset network model to obtain an image processing model.

[0103] Specifically, an optical camera is used to capture a visible light image of the area to be measured. c Its resolution is (w c ,h c Low-resolution infrared images of the area to be measured are acquired using low-resolution infrared sensors (such as infrared array sensors). il As shown in the first sample infrared image above, its resolution is (w il ,h il High-resolution infrared images of the area to be measured are acquired using a high-resolution infrared sensor. ih As shown in the second sample infrared image above, its resolution is (w ih ,h ih ( ). Among them, because the photosensitive devices of optical cameras are less expensive, higher resolution optical cameras can be used, with a maximum resolution that can reach 10-100 times that of infrared sensors. Therefore, w c ≥w ih >w il ,h c ≥h ih h il Visible light images have high resolution and contain detailed information about objects in the target field of view, such as contour information and boundary information. They can help to further expand and refine the temperature information of each pixel in low-resolution infrared images, thereby improving the resolution of infrared images and obtaining high-resolution infrared images.

[0104] By acquiring several sets of preset sample images, and using the first sample infrared image, the second sample infrared image, and the sample visible light image from each set, a preset network model can be trained to obtain an image processing model. This image processing model is used to output the corresponding second target infrared image based on the first target infrared image and the target visible light image of any target region, thus improving the resolution of the infrared image.

[0105] The sample dataset contains several sets of preset sample images. Each set includes a first sample infrared image acquired by a low-resolution infrared sensor, a second sample infrared image acquired by a high-resolution sensor, and a sample visible light image acquired by a high-resolution optical camera, all within the same field of view. Furthermore, the preset sample images in the dataset can be divided into a training set, a validation set, and a test set in a 3:1:1 ratio for training, validation, and testing of the neural network, respectively.

[0106] In this scheme, an image processing model is trained to combine low-resolution infrared images obtained at low cost with visible light images to obtain high-resolution infrared images. This saves costs while expanding the information of infrared images, and accurately and efficiently improves the resolution of infrared images.

[0107] In a feasible solution, such as Figure 2 As shown, step S2 includes:

[0108] S201. Input the first sample infrared image and sample visible light image of each group into the preset network model, and output the simulated infrared image;

[0109] The resolution of the simulated infrared image is higher than that of the first sample infrared image;

[0110] S202. Calculate the loss value of the simulated infrared image compared to the corresponding second sample infrared image;

[0111] S203. Update the model parameters of the preset network model based on the loss value until the loss value meets the preset loss condition to obtain the image processing model.

[0112] Specifically, the first sample infrared image and the sample visible light image from each set of preset sample images are input into a preset network model, and a simulated infrared image is output. A loss value is calculated between the simulated infrared image and the corresponding second sample infrared image based on a preset loss function. The model parameters of the preset network model are updated based on this loss value to obtain the image processing model. The cross-entropy of the RGB (red, green, and blue) values ​​of each pixel is used as the loss function to train the preset network model to minimize the error, and the model parameters at that time are saved.

[0113] In this scheme, an image processing model is trained using preset sample images. The model parameters are updated by backpropagating the loss value between the simulated infrared image and the corresponding second sample infrared image. This improves the performance of the image processing model and thus enhances the accuracy of the model when outputting high-resolution infrared images.

[0114] In a feasible solution, such as Figure 3As shown, step S201 includes:

[0115] S2011. For each set of preset sample images input to the preset network model, obtain several infrared feature maps at a first resolution based on the first sample infrared image.

[0116] S2012. Based on the sample visible light image, obtain several optical feature maps at the second resolution.

[0117] S2013. Each infrared feature map is stitched together with an optical feature map to obtain several fused feature maps at the third resolution.

[0118] Among them, the resolution difference between the spliced ​​infrared feature map and the optical feature map is less than the first preset resolution difference;

[0119] S2014. Obtain a simulated infrared image based on several fused feature maps.

[0120] Specifically, infrared feature maps and optical feature maps are first obtained through feature extraction; then, a fused feature map is obtained by stitching the two feature maps together, wherein the number of channels in the fused feature map is the sum of the number of channels in the infrared feature map and the optical feature map; finally, the fused feature map is subjected to reconstruction operations such as convolution, downsampling, and stitching to obtain a high-resolution simulated infrared image.

[0121] In this scheme, by stitching and fusing infrared feature maps and optical feature maps, the amount of information in the infrared feature maps can be expanded, thereby improving the accuracy of the simulated infrared images output by the model.

[0122] In one feasible solution, the pre-defined network model includes an infrared feature extraction module, an optical feature extraction module, a stitching module, and an image reconstruction module;

[0123] The infrared feature extraction module is used to acquire infrared feature maps;

[0124] The optical feature extraction module is used to acquire optical feature maps;

[0125] The stitching module is used to stitch together infrared feature maps and optical feature maps;

[0126] The image reconstruction module is used to acquire simulated infrared images.

[0127] In this scheme, a preset network model is constructed through an infrared feature extraction module, an optical feature extraction module, a stitching module, and an image reconstruction module to train an image processing model, which can improve the model's performance and thus improve the accuracy of the model when outputting high-resolution infrared images.

[0128] In one feasible solution, the network model is assumed to be a convolutional neural network model;

[0129] Step S2011 includes: alternatingly performing convolution and upsampling processing on the first sample infrared image to obtain several infrared feature maps at a first resolution;

[0130] And / or,

[0131] Step S2012 includes: alternatingly performing convolution and pooling processing on the sample visible light image to obtain several optical feature maps at a second resolution;

[0132] And / or,

[0133] Step S2014 includes: alternating convolution and upsampling processing on the fused feature map to obtain a simulated infrared image.

[0134] Specifically, since high-resolution feature maps can preserve the detailed information of an image, while low-resolution feature maps can convey the semantic information of the image, convolutional processing is alternated with pooling or upsampling processing during the feature extraction stage to obtain infrared and visible light feature maps at different resolutions. For visible light images, through operations such as convolution and pooling, the feature map size (i.e., resolution) continuously decreases while the number of channels continuously increases; for infrared images, through operations such as convolution and upsampling, both the feature map size and the number of channels continuously increase.

[0135] In the feature fusion stage, infrared and optical feature maps of similar or identical size can be stitched together, allowing the features of the two input images to be fused. Similar feature map sizes indicate that the two heterogeneous feature maps have similar scale of feature information, making them suitable for fusion into a single entity.

[0136] During the reconstruction phase, one of the fused feature maps is alternately convolved and upsampled, and other fused feature maps are continuously stitched together to obtain a simulated infrared image.

[0137] In this scheme, by acquiring infrared and optical feature maps with different resolutions, the detailed information of the image and the semantic information of the image can be preserved. Furthermore, by stitching and fusing infrared and optical feature maps with resolution differences within a preset range, the information content of the infrared feature map can be expanded, thereby improving the accuracy of the high-resolution infrared image output by the model.

[0138] In one feasible embodiment, step S2014 includes:

[0139] The fusion feature map with the smallest resolution is alternately processed by convolution and upsampling. After each upsampling process, another fusion feature map is stitched together. The process is repeated in order of increasing resolution until all fusion feature maps are traversed to obtain a simulated infrared image.

[0140] In each stitch, the resolution difference between the two fused feature maps is less than the second preset resolution difference.

[0141] Specifically, the fused feature map with the lowest resolution is alternately subjected to convolution and downsampling processes, and the feature map obtained by the (i-1)th downsampling process is concatenated with the fused feature map with a similar resolution (the difference is less than the second preset resolution difference) to serve as the input feature map for the i-th convolution process.

[0142] During the reconstruction phase, a multi-level upsampling method is used to increase the resolution of the feature map. During the upsampling process, fused feature maps of different resolutions from the feature fusion phase are continuously stitched together, so that the features have continuity and stability.

[0143] In this scheme, by continuously stitching together fused feature maps of different resolutions during the convolution and upsampling processes, the continuity and stability of the image features are ensured, while taking into account both the detailed and overall features of the image, thereby improving the accuracy of the output high-resolution infrared image.

[0144] In one feasible approach, the step of stitching each infrared feature map with an optical feature map includes the following prior to the following:

[0145] At least one feature map from the stitched infrared feature map and optical feature map is cropped and / or expanded to make the resolution of the infrared feature map and the optical feature map equal.

[0146] And / or,

[0147] The step of stitching together another fused feature map after each upsampling process also includes:

[0148] At least one of the two fused feature maps is cropped and / or expanded to make the two fused feature maps have the same resolution.

[0149] Specifically, when the size of the spliced ​​feature maps is different, edge padding or expansion operations can be performed on the perimeter of one or two feature maps. Taking expansion as an example, common padding values ​​include 0, 0.5, and 1 (the padding value in this embodiment is 0). If the size of the feature map is 244x244 pixels and needs to be expanded to 280x280 pixels, then (280-244) / 2 = 18 pixels of 0 value are added to each of the four sides of the original feature map, making the feature map size 280x280.

[0150] In this scheme, the feature maps are cropped or expanded to ensure that the resolution of several feature maps to be stitched is equal, thus guaranteeing the accuracy of feature stitching and fusion.

[0151] The following is combined Figure 4 The specific structure of the image processing model trained by the training method in this embodiment is explained below:

[0152] This image processing model mainly consists of three parts, used for feature extraction, feature fusion, and high-resolution infrared image generation:

[0153] (1) In the feature extraction stage, the input infrared temperature field image with a resolution of 68*68 and a number of channels is subjected to several convolution and pooling processes alternately to obtain several infrared feature maps with different resolutions and different numbers of channels; the input visible light image with a resolution of 572*572 and a number of channels is subjected to several convolution and upsampling processes alternately to obtain several optical feature maps with different resolutions and different numbers of channels.

[0154] (2) In the feature fusion stage, infrared feature maps and optical feature maps of similar or identical size are stitched together to obtain a fused feature map (e.g., Figure 4 The features of the two input images (A, B, and C) are fused together. The number of channels in the fused feature map is the sum of the number of channels in the infrared feature map and the optical feature map.

[0155] (3) In the high-resolution infrared image generation stage, a multi-level upsampling method is used to increase the resolution of the feature map. During the upsampling process, fused feature maps of different resolutions from the feature fusion stage are continuously stitched together, so that the features have continuity and stability. Specifically, for the smallest resolution fused feature map ( Figure 4 A fused feature map with a medium resolution of 64*64 and 576 channels is alternately processed by convolution and downsampling. The feature map obtained by the (i-1)th downsampling process is then concatenated with a fused feature map of similar resolution (the difference is less than the second preset resolution difference). Figure 4 The feature maps A', B', and C' (separated from each other) are used as the input feature maps for the i-th convolution process. A corresponds to A', where A is the fused feature map generated in the feature fusion stage and is used as the input for the final image generation stage, concatenated with the upsampled result. Since the two input feature maps have different sizes during concatenation (e.g., A is 136x136, and the object to be concatenated is 120x120), each of the four sides of A is cropped by 8 pixels to achieve a size of 120x120, hence denoted as A'. B and B', and C and C' have the same meaning. Finally, a convolution operation is performed on the feature map with a resolution of 456*456 and 1152 channels, restoring its channel count to 3 while maintaining high resolution, outputting a high-resolution infrared image.

[0156] The image processing model training method provided in this embodiment can combine low-resolution infrared images obtained at low cost with visible light images to obtain high-resolution infrared images, thus expanding the information of infrared images while saving costs. Through multiple levels of convolution, pooling and upsampling processing, it takes into account both the detailed features and the overall features of the image, ensuring the accuracy of the output high-resolution infrared image, thereby accurately and efficiently improving the resolution of infrared images.

[0157] Example 2

[0158] This embodiment provides an image processing method, which is implemented based on the image processing model obtained by the training method in Embodiment 1, such as... Figure 5 As shown, the image processing method includes the following steps:

[0159] S3. Acquire the infrared image and visible light image of the first target from the same target perspective;

[0160] S4. Input the infrared image of the first target and the visible light image of the target into the image processing model to output the infrared image of the second target;

[0161] The resolution of the infrared image of the second target is higher than that of the infrared image of the first target.

[0162] The image processing method provided in this embodiment can combine a low-cost first target infrared image with a target visible light image to obtain a high-resolution second target infrared image through an image processing model. This saves costs while expanding the information of the infrared image, ensuring the accuracy of the output high-resolution infrared image, and thus accurately and efficiently improving the resolution of the infrared image.

[0163] Example 3

[0164] This embodiment provides a training system for an image processing model, such as... Figure 6 As shown, the training system includes:

[0165] Sample image acquisition module 1 is used to acquire several sets of preset sample images;

[0166] Each set of preset sample images includes a first sample infrared image, a second sample infrared image, and a sample visible light image from the same sample viewpoint. The resolution of the first sample infrared image is lower than that of the second sample infrared image and the sample visible light image.

[0167] Model training module 2 is used to train a preset network model based on each set of preset sample images to obtain an image processing model.

[0168] In a feasible solution, such as Figure 7 As shown, model training module 2 includes:

[0169] The simulated image output unit 21 is used to input the first sample infrared image and sample visible light image of each group into a preset network model and output a simulated infrared image.

[0170] The resolution of the simulated infrared image is higher than that of the first sample infrared image;

[0171] The loss calculation unit 22 is used to calculate the loss value of the simulated infrared image compared to the corresponding second sample infrared image;

[0172] The parameter update unit 23 is used to update the model parameters of the preset network model based on the loss value until the loss value meets the preset loss condition to obtain the image processing model.

[0173] In one feasible embodiment, the analog image output unit 21 is further configured to:

[0174] For each set of preset sample images input to the preset network model, several infrared feature maps at a first resolution are obtained based on the first sample infrared image.

[0175] Based on the sample visible light image, obtain several optical feature maps at a second resolution;

[0176] Each infrared feature map is stitched together with an optical feature map to obtain several fused feature maps at the third resolution.

[0177] Among them, the resolution difference between the spliced ​​infrared feature map and the optical feature map is less than the first preset resolution difference;

[0178] A simulated infrared image is obtained based on several fused feature maps.

[0179] In one feasible solution, the pre-defined network model includes an infrared feature extraction module, an optical feature extraction module, a stitching module, and an image reconstruction module;

[0180] The infrared feature extraction module is used to acquire infrared feature maps;

[0181] The optical feature extraction module is used to acquire optical feature maps;

[0182] The stitching module is used to stitch together infrared feature maps and optical feature maps;

[0183] The image reconstruction module is used to acquire simulated infrared images.

[0184] In one feasible solution, the network model is assumed to be a convolutional neural network model;

[0185] The analog image output unit 21 is also used for:

[0186] The first sample infrared image is alternately convolved and upsampled to obtain several infrared feature maps at the first resolution.

[0187] And / or,

[0188] The sample visible light images are alternately convolutional and pooling to obtain several optical feature maps at a second resolution.

[0189] And / or,

[0190] The fused feature map is alternately convolved and upsampled to obtain a simulated infrared image.

[0191] In one feasible embodiment, the analog image output unit 21 is further configured to:

[0192] The fusion feature map with the smallest resolution is alternately processed by convolution and upsampling. After each upsampling process, another fusion feature map is stitched together. The process is repeated in order of increasing resolution until all fusion feature maps are traversed to obtain a simulated infrared image.

[0193] In each stitch, the resolution difference between the two fused feature maps is less than the second preset resolution difference.

[0194] In one feasible embodiment, the analog image output unit 21 is further configured to:

[0195] At least one feature map from the stitched infrared feature map and optical feature map is cropped and / or expanded to make the resolution of the infrared feature map and the optical feature map equal.

[0196] And / or,

[0197] At least one of the two fused feature maps is cropped and / or expanded to make the two fused feature maps have the same resolution.

[0198] Since the training system of the image processing model provided in this embodiment is based on the same principle as the training method of the image processing model provided in Embodiment 1, it will not be described again here.

[0199] The image processing model training system provided in this embodiment can combine low-resolution infrared images obtained at low cost with visible light images to obtain high-resolution infrared images, thus expanding the information of infrared images while saving costs. Through multiple levels of convolution, pooling and upsampling processing, it takes into account both the detailed features and the overall features of the image, ensuring the accuracy of the output high-resolution infrared image, thereby accurately and efficiently improving the resolution of infrared images.

[0200] Example 4

[0201] This embodiment provides an image processing system, which is implemented based on the image processing model obtained from the training system in Embodiment 3, such as... Figure 8 As shown, the image processing system includes:

[0202] Target image acquisition module 3 is used to acquire the infrared image and the visible light image of the first target from the same target viewpoint;

[0203] Image processing module 4 is used to input the infrared image of the first target and the visible light image of the target into the image processing model to output the infrared image of the second target;

[0204] The resolution of the infrared image of the second target is higher than that of the infrared image of the first target.

[0205] Since the image processing system provided in this embodiment is based on the same principle as the image processing method provided in embodiment 2, it will not be described again here.

[0206] The image processing method provided in this embodiment can combine a low-cost first target infrared image with a target visible light image to obtain a high-resolution second target infrared image through an image processing model. This saves costs while expanding the information of the infrared image, ensuring the accuracy of the output high-resolution infrared image, and thus accurately and efficiently improving the resolution of the infrared image.

[0207] Example 5

[0208] This embodiment provides an electronic device. Figure 9 This is a schematic diagram of the electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the training method of the image processing model in Embodiment 1 and the image processing method in Embodiment 2. Figure 9 The electronic device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0209] like Figure 9 As shown, the electronic device 30 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 32 and processor 31).

[0210] Bus 33 includes a data bus, an address bus, and a control bus.

[0211] The memory 32 may include volatile memory, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0212] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0213] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the image processing model training method of Embodiment 1 and the image processing method of Embodiment 2 of this disclosure.

[0214] Electronic device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 35. Furthermore, the model-generating device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 36. Figure 9 As shown, network adapter 36 communicates with other modules of the model-generated device 30 via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0215] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0216] Example 6

[0217] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method of the image processing model of Embodiment 1 and the image processing method of Embodiment 2.

[0218] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0219] In possible implementations, this disclosure can also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to execute the training method for the image processing model of Embodiment 1 and the image processing method of Embodiment 2.

[0220] The program code for executing this disclosure can be written in any combination of one or more programming languages. The program code can be executed entirely on a user device, partially on a user device, as a standalone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0221] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A method for training an image processing model, characterized in that, The training method comprises: obtaining a plurality of groups of preset sample images; each group of the preset sample images comprises a first sample infrared image, a second sample infrared image and a sample visible light image under the same sample view angle, the resolution of the first sample infrared image is lower than the resolution of the second sample infrared image and the resolution of the sample visible light image; training a preset network model based on each group of the preset sample images to obtain the image processing model; the step of training the preset network model based on each group of the preset sample images to obtain the image processing model comprises: inputting the first sample infrared image and the sample visible light image of each group into the preset network model to output a simulated infrared image; the resolution of the simulated infrared image is higher than the resolution of the first sample infrared image; calculating a loss value of the simulated infrared image compared with the corresponding second sample infrared image; updating the model parameters of the preset network model based on the loss value until the loss value meets a preset loss condition to obtain the image processing model; the step of inputting the first sample infrared image and the sample visible light image of each group into the preset network model to output a simulated infrared image comprises: for each group of the preset sample images input into the preset network model, obtaining a plurality of infrared feature maps under a first resolution according to the first sample infrared image; obtaining a plurality of optical feature maps under a second resolution according to the sample visible light image; splicing each infrared feature map with an optical feature map to obtain a plurality of fusion feature maps under a third resolution; the resolution difference between the spliced infrared feature map and the optical feature map is less than a first preset resolution difference; obtaining the simulated infrared image according to a plurality of the fusion feature maps. 2.The method of claim 1, wherein, The preset network model comprises an infrared feature extraction module, an optical feature extraction module, a splicing module and an image reconstruction module; the infrared feature extraction module is used to obtain the infrared feature map; the optical feature extraction module is used to obtain the optical feature map; the splicing module is used to splice the infrared feature map and the optical feature map; the image reconstruction module is used to obtain the simulated infrared image. 3.The method of claim 1, wherein, The preset network model is a convolutional neural network model; the step of obtaining a plurality of infrared feature maps under a first resolution according to the first sample infrared image comprises: alternately performing convolution processing and up-sampling processing on the first sample infrared image to obtain a plurality of infrared feature maps under the first resolution; and / or the step of obtaining a plurality of optical feature maps under a second resolution according to the sample visible light image comprises: alternately performing convolution processing and pooling processing on the sample visible light image to obtain a plurality of optical feature maps under the second resolution; and / or the step of obtaining the simulated infrared image according to a plurality of the fusion feature maps comprises: alternately performing convolution processing and up-sampling processing on the fusion feature map to obtain the simulated infrared image. 4.The method of Claim 3, wherein The step of alternately performing convolution processing and up-sampling processing on the fusion feature maps to obtain the simulated infrared image comprises: alternately performing convolution processing and up-sampling processing on the fusion feature map with the smallest resolution, and splicing another fusion feature map after each up-sampling processing, sequentially traversing and processing in order from small to large resolution, until all the fusion feature maps are traversed, to obtain the simulated infrared image. Wherein, the resolution difference between the two fusion feature maps spliced each time is less than a second preset resolution difference. 5.The method of Claim 4, wherein The step of splicing each infrared feature map with an optical feature map further comprises: performing clipping processing and / or expansion processing on at least one of the spliced infrared feature map and the optical feature map, so that the resolutions of the infrared feature map and the optical feature map are equal; and / or, The step of splicing another fusion feature map after each up-sampling processing further comprises: performing clipping processing and / or expansion processing on at least one of the two spliced fusion feature maps, so that the resolutions of the two fusion feature maps are equal.

6. A system for training an image processing model, the system comprising: The training system comprises: a sample image acquisition module for acquiring a plurality of groups of preset sample images; Wherein, each group of preset sample images comprises a first sample infrared image, a second sample infrared image and a sample visible light image under the same sample view angle, the resolution of the first sample infrared image is lower than the resolution of the second sample infrared image and the resolution of the sample visible light image; a model training module for training a preset network model based on each group of preset sample images to obtain the image processing model; The model training module comprises: a simulated image output unit for inputting the first sample infrared image and the sample visible light image of each group into the preset network model, and outputting a simulated infrared image; Wherein, the resolution of the simulated infrared image is higher than the resolution of the first sample infrared image; a loss calculation unit for calculating a loss value of the simulated infrared image compared with the corresponding second sample infrared image; a parameter updating unit for updating the model parameters of the preset network model based on the loss value until the loss value meets a preset loss condition, to obtain the image processing model; The simulated image output unit is further configured to: for each group of preset sample images input into the preset network model, acquire infrared feature maps at a plurality of first resolutions according to the first sample infrared image; acquire optical feature maps at a plurality of second resolutions according to the sample visible light image; splicing each infrared feature map with an optical feature map to obtain fusion feature maps at a plurality of third resolutions; Wherein, the resolution difference between the spliced infrared feature map and the optical feature map is less than a first preset resolution difference; acquire the simulated infrared image according to a plurality of fusion feature maps.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory for running on the processor, characterized in that, The processor executes the computer program to implement the training method of the image processing model of any one of claims 1-5. The processor executes the computer program to implement the training method of the image processing model of any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the training method of the image processing model according to any one of claims 1-5.

Images