An infrared image generation method, device, equipment and medium
By processing and fusing the low-frequency and high-frequency spatial features of visible light images, and using a neural network model to generate infrared images, the problem of the single method of infrared image collection is solved, and diversified generation of infrared images is realized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2022-12-07
- Publication Date
- 2026-06-05
Smart Images
Figure CN115861753B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device and medium for generating infrared images. Background Technology
[0002] In photoelectric images across different wavelengths, the visible light band generates visible light images, while the infrared band generates infrared images. Visible light images are formed by light within the wavelength range visible to the human eye (e.g., 0.38–0.78 μm), primarily reflecting the distribution of reflected energy on the surface of the photographed scene. Infrared images, on the other hand, are formed by light within the infrared band, specifically by infrared radiation reflected from objects or the target itself by the infrared imaging system. They primarily reflect the infrared radiation characteristics of the photographed target and can effectively characterize the distribution of radiant energy in the photographed scene.
[0003] Infrared imaging technology, with its advantages of high guidance accuracy, strong anti-interference ability, long-distance imaging, and night imaging, can better realize infrared target detection, monitoring, tracking, and weapon guidance.
[0004] In real-world scenarios, infrared images are mostly collected on-site using various infrared cameras. This method of collecting infrared images is relatively simple and cannot meet the large demand for infrared images. Summary of the Invention
[0005] This application provides a method, apparatus, device, and medium for generating infrared images, which provides a way to generate infrared images based on visible light images, offering more ways to collect infrared images and meeting the large demand for infrared images to a certain extent.
[0006] In a first aspect, this application provides a method for generating an infrared image, comprising:
[0007] Determine the low-frequency and high-frequency spatial features corresponding to the visible light image;
[0008] Based on the low-frequency spatial features and the high-frequency spatial features, a first feature of the visible light image is determined, which is used to represent the transparency of different regions in the visible light image.
[0009] The high-frequency spatial features are enhanced, and the first feature is used for mask-like processing to obtain a second feature. The second feature is used to represent the high-frequency texture of the highly transparent region in the visible light image.
[0010] The low-frequency spatial features and the second feature are fused to determine the fused image features;
[0011] The fused image features are input into a pre-trained neural network model to obtain an infrared image corresponding to the visible light image. The neural network model is used to generate an infrared image based on the input image.
[0012] In one possible implementation, before fusing the low-frequency spatial features and the second feature to determine the fused image features, the method further includes:
[0013] Based on the aforementioned low-frequency spatial features, a low-frequency spatial feature mask is determined;
[0014] The second feature is processed using the low-frequency spatial feature mask to obtain a third feature, which is used to represent the high-frequency texture of the highly transparent region in the visible light image.
[0015] The low-frequency spatial features and the second feature are fused to determine the fused image features, including:
[0016] The low-frequency spatial features and the third feature are fused to determine the fused image features.
[0017] In one possible implementation, before fusing the low-frequency spatial features and the second feature to determine the fused image features, the method further includes:
[0018] Determine the mid-frequency spatial features corresponding to the visible light image;
[0019] The mid-frequency spatial domain feature is weakened and the first feature is used for mask-like processing to obtain a fourth feature, which is used to represent the mid-frequency texture of the highly transparent region in the visible light image.
[0020] The low-frequency spatial features and the second feature are fused to determine the fused image features, including:
[0021] The low-frequency spatial features, the second feature, and the fourth feature are fused to determine the fused image features.
[0022] In one possible implementation, before fusing the low-frequency spatial features, the second feature, and the fourth feature to determine the fused image features, the method further includes:
[0023] Based on the aforementioned low-frequency spatial features, a low-frequency spatial feature mask is determined;
[0024] The fourth feature is processed using the low-frequency spatial feature mask to obtain the fifth feature, which is used to represent the high-frequency texture of the highly transparent region in the visible light image.
[0025] The low-frequency spatial features, the second feature, and the fourth feature are fused to determine the fused image features, including:
[0026] The low-frequency spatial features, the second feature, and the fifth feature are fused to determine the fused image features.
[0027] In one possible implementation, before fusing the low-frequency spatial features and the third feature to determine the fused image features, the method further includes:
[0028] Determine the mid-frequency spatial features corresponding to the visible light image;
[0029] The mid-frequency spatial domain feature is weakened and the first feature is used for mask-like processing to obtain a fourth feature, which is used to represent the mid-frequency texture of the highly transparent region in the visible light image.
[0030] The low-frequency spatial features and the third feature are fused to determine the fused image features, including:
[0031] The low-frequency spatial features, the third feature, and the fourth feature are fused to determine the fused image features.
[0032] In one possible implementation, before fusing the low-frequency spatial features, the third feature, and the fourth feature to determine the fused image features, the method further includes:
[0033] The fourth feature is processed using the low-frequency spatial feature mask to obtain the fifth feature, which is used to represent the mid-frequency texture of the highly transparent region in the visible light image.
[0034] The low-frequency spatial features, the third feature, and the fourth feature are fused to determine the fused image features, including:
[0035] The low-frequency spatial features, the third feature, and the fifth feature are fused to determine the fused image features.
[0036] Secondly, this application provides an infrared image generation apparatus, comprising:
[0037] The spatial domain feature module is used to determine the low-frequency and high-frequency spatial domain features corresponding to the visible light image;
[0038] A transparency module is used to determine a first feature of the visible light image based on the low-frequency spatial features and the high-frequency spatial features, wherein the first feature represents the transparency of different regions in the visible light image; enhance the high-frequency spatial features and perform mask-like processing using the first feature to obtain a second feature, wherein the second feature represents the high-frequency texture of regions with high transparency in the visible light image;
[0039] The fusion module is used to fuse the low-frequency spatial features and the second feature to determine the fused image features;
[0040] An infrared module is used to input the fused image features into a pre-trained neural network model to obtain an infrared image corresponding to the visible light image.
[0041] One possible implementation also includes:
[0042] A mask module is used to determine a low-frequency spatial feature mask based on the low-frequency spatial features; the second feature is processed using the low-frequency spatial feature mask to obtain a third feature, which is used to represent the high-frequency texture of the region with high transparency in the visible light image;
[0043] The fusion module is specifically used to fuse the low-frequency spatial features and the third feature to determine the fused image features.
[0044] In one possible implementation, the spatial domain feature module is further configured to determine the mid-frequency spatial domain features corresponding to the visible light image;
[0045] The transparency module is also used to weaken the mid-frequency spatial features and perform mask-like processing using the first feature to obtain a fourth feature, which is used to represent the mid-frequency texture of the region with high transparency in the visible light image.
[0046] The fusion module is specifically used to fuse the low-frequency spatial features, the second feature, and the fourth feature to determine the fused image features.
[0047] One possible implementation also includes:
[0048] A mask module is used to determine a low-frequency spatial feature mask based on the low-frequency spatial features; the low-frequency spatial feature mask is used to process the fourth feature to obtain a fifth feature, which is used to represent the high-frequency texture of the region with high transparency in the visible light image;
[0049] The fusion module is specifically used to fuse the low-frequency spatial features, the second feature, and the fifth feature to determine the fused image features.
[0050] In one possible implementation, the spatial domain feature module is further configured to determine the mid-frequency spatial domain features corresponding to the visible light image;
[0051] The transparency module is also used to weaken the mid-frequency spatial features and perform mask-like processing using the first feature to obtain a fourth feature, which is used to represent the mid-frequency texture of the region with high transparency in the visible light image.
[0052] The fusion module is specifically used to fuse the low-frequency spatial features, the third feature, and the fourth feature to determine the fused image features.
[0053] One possible implementation also includes:
[0054] A mask module is used to process the fourth feature using the low-frequency spatial feature mask to obtain a fifth feature, which is used to represent the mid-frequency texture of the region with high transparency in the visible light image;
[0055] The fusion module is specifically used to fuse the low-frequency spatial features, the third feature, and the fifth feature to determine the fused image features.
[0056] Thirdly, this application provides an electronic device, including: a processor, optionally further including a memory; the processor and the memory are coupled; the memory is used to store computer programs or instructions; the processor is used to execute part or all of the computer programs or instructions in the memory, and when the part or all of the computer programs or instructions are executed, it is used to implement the function in any of the above methods.
[0057] In one possible implementation, the apparatus may further include a transceiver for transmitting signals processed by the processor or receiving signals input to the processor. The transceiver may perform either the transmitting or receiving action of any of the methods.
[0058] Fourthly, a computer-readable storage medium is provided for storing a computer program, the computer program including instructions for implementing any of the functions.
[0059] Alternatively, a computer-readable storage medium for storing a computer program, which, when executed by a computer, causes the computer to perform any of the methods described above.
[0060] Fifthly, a computer program product is provided, the computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform any of the methods described above.
[0061] In this embodiment, a fused image is generated based on the low-frequency and high-frequency spatial features of a visible light image. This fused image is then input into a pre-trained neural network model to obtain an infrared image corresponding to the visible light image, thus realizing the generation of a fused image based on a visible light image. Since low-frequency signals can represent the content or type of a visible light image, low-frequency spatial features must be preserved during the generation of the fused image. Since high-frequency signals can represent the texture portion of a visible light image, the high-frequency spatial features are enhanced during the generation of the fused image. Furthermore, by enhancing the high-frequency spatial features and using the first feature for mask-like processing, the high-frequency texture features in highly transparent areas of the visible light image can be enhanced, highlighting the texture features. In this way, generating a visible light image based on the fused image not only preserves the content or type information of the image but also, through the superposition of enhanced high-frequency texture features, makes the infrared image generated from the fused image closely resemble a real infrared image. In addition, the entire generation process is simple and fast. This embodiment provides more methods for collecting infrared images and can, to a certain extent, meet the large demand for infrared images. Attached Figure Description
[0062] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.
[0063] Figure 1 A schematic diagram of an infrared image generation process provided in an embodiment of this application is shown;
[0064] Figure 2 A schematic diagram of an infrared image generation process provided in an embodiment of this application is shown;
[0065] Figure 3 This paper illustrates a structural diagram of an infrared image generation apparatus provided in an embodiment of this application.
[0066] Figure 4 A structural diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation
[0067] To make the objectives and implementation methods of this application clearer, exemplary embodiments of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments. Obviously, the described exemplary embodiments are only a part of the embodiments of this application, and not all of the embodiments. The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms can be used interchangeably where appropriate.
[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
[0069] The following explains the technical terms or nouns used in this application:
[0070] 1) High and low frequencies in an image are a measure of the intensity variation between different locations within the image. High-frequency signals in an image refer to areas where the image intensity (brightness / grayscale) changes drastically, such as edges (contours) and textures. Low-frequency signals refer to areas where the image intensity (brightness / grayscale) changes gently, i.e., large areas of color, usually indicating the content or type of the image, such as cats or dogs. The human eye is more sensitive to high-frequency signals in images. For example, on a white sheet of paper, the text is a high-frequency signal, while the white paper is a low-frequency signal. Similarly, in a facial image, blemishes and wrinkles are more varied than a clean face, and therefore represent high-frequency signals; this can be understood as the internal texture of the facial image.
[0071] 2) The difference between mask-like processing and mask processing includes: the pixel values in the image features used in mask-like processing (such as the first feature mentioned later) include not only 0 and 1, but also other values between 0 and 1.
[0072] 3) Definitions in the embodiments of this application: Represents the low-frequency spatial characteristic matrix. Represents the mid-frequency spatial domain characteristic matrix. Represents the high-frequency spatial characteristic matrix. Denotes the first characteristic matrix. This represents the second characteristic matrix. This represents the fourth characteristic matrix. Let F represent the low-frequency spatial feature mask matrix, and let F represent the fused image feature matrix. For ease of description, the word "matrix" will be omitted thereafter.
[0073] Figure 1 The illustration shows a schematic diagram of an infrared image generation process provided by an embodiment of this application. This method can be applied to electronic devices, such as monitoring devices or other devices capable of image processing.
[0074] The process includes the following steps:
[0075] Step 101: Determine the low-frequency spatial features corresponding to the visible light image. and high-frequency spatial characteristics .
[0076] Step 101 will be explained in detail later.
[0077] Step 102: Based on the aforementioned low-frequency spatial characteristics and the aforementioned high-frequency spatial features Determine the first feature of the visible light image. The first feature is used to represent the transparency of different regions in the visible light image.
[0078] The characteristic differences between low and high frequencies can, to some extent, indicate the transparency of an object.
[0079] The low-frequency features of an image retain most of the original image content. Based on the low-frequency features, the type and outline of objects in the image can be determined. The type can estimate whether the object is transparent, and the outline can determine the transparency range of the object. On the other hand, the high-frequency spatial features retain the high-frequency texture of the image. If there are texture features in the high-frequency part that are inconsistent with the type of objects in the image, then after processing, the high-frequency texture features and low-frequency object appearance features in the image can, to a certain extent, characterize the transparency of the corresponding image region.
[0080] For example, based on the following formula: The first feature of the visible light image is determined.
[0081] For example, it can be used to analyze low-frequency spatial characteristics. and the aforementioned high-frequency spatial features After normalization, the first feature of the visible light image is determined. Thus, the value of the first feature ranges from 0 to 1, where values closer to 0 indicate opacity, and values closer to 1 indicate transparency.
[0082] For example, based on the following formula: Determine the first feature of the visible light image; wherein, .
[0083] In addition, in this formula All functions can be replaced with the tanh function, as well as those discussed later. All functions can be replaced with the tanh function. tanh function and The function is used to control the reference value between 0 and 1.
[0084] Furthermore, redundant information in low-frequency and / or high-frequency spatial features can be removed before determining the first feature of the visible light image. This reduces the complexity of subsequent use of low-frequency and / or high-frequency spatial features. In one example, the spatial features are input into a neural network model, which can remove redundant information from the spatial features.
[0085] For example, based on the following formula: Determine the first feature of a visible light image. .
[0086] For example, based on the following formula: Determine the first feature of a visible light image. .
[0087] For example, based on the following formula: Determine the first feature of a visible light image. .
[0088] For example, based on the following formula: Determine the first feature of a visible light image. .
[0089] For example, based on the following formula: Determine the first feature of a visible light image. .
[0090] For example, based on the following formula: Determine the first feature of a visible light image. .
[0091] in, Represents a neural network model. This represents a neural network model. This indicates that the input spatial features x are processed by several layers of neural networks, which can remove redundant information from the input spatial features x. , and subsequent mentions , , , , They can be the same or different.
[0092] Step 103: Analyze the high-frequency spatial features Enhancement processing is performed, and the first feature is employed. Perform masking to obtain the second feature. .
[0093] The second feature can be used to characterize the high-frequency texture features of highly transparent areas in a visible light image.
[0094] The areas with high transparency can be understood as the areas in P where the value is greater than a set threshold.
[0095] For example, the enhancement processing of the high-frequency spatial features is as follows: , A value greater than or equal to 2. High-frequency signals can represent the texture in visible light images, and these texture features can be highlighted by enhancing the high-frequency spatial features.
[0096] For example, based on the following formula: Determine the second feature; among which, A value greater than or equal to 2.
[0097] Furthermore, a neural network model can be used first. High-frequency spatial characteristics After removing redundant information, enhancement processing is performed, and the first feature is used. Perform a masking process to obtain the second feature. For example, based on the following formula: , determine the second feature.
[0098] Step 104: The low-frequency spatial features and the second feature Perform fusion to determine the fused image features F.
[0099] Fusion can be understood as superposition.
[0100] For example, based on the following formula (1): F To determine the features of the fused image.
[0101] Furthermore, a neural network model can be used first. After removing redundant information from the low-frequency spatial features, then the second feature The fusion process is performed to determine the features of the fused image. For example, based on the following formula (2): F To determine the features of the fused image.
[0102] Step 105: Input the fused image features F into a pre-trained neural network model to obtain an infrared image corresponding to the visible light image.
[0103] In this embodiment, a fused image is generated based on the low-frequency and high-frequency spatial features of a visible light image. This fused image is then input into a pre-trained neural network model to obtain an infrared image corresponding to the visible light image, thus realizing the generation of a fused image based on a visible light image. Since low-frequency signals can represent the content or type of a visible light image, low-frequency spatial features must be preserved during the generation of the fused image. Since high-frequency signals can represent the texture portion of a visible light image, the high-frequency spatial features are enhanced during the generation of the fused image. Furthermore, by enhancing the high-frequency spatial features and using the first feature for mask-like processing, the high-frequency texture features in highly transparent areas of the visible light image can be enhanced, highlighting the texture features. In this way, generating a visible light image based on the fused image not only preserves the content or type information of the image but also, through the superposition of enhanced high-frequency texture features, makes the infrared image generated from the fused image closely resemble a real infrared image. In addition, the entire generation process is simple and fast. This embodiment provides more methods for collecting infrared images and can, to a certain extent, meet the large demand for infrared images.
[0104] For example, features of infrared images .
[0105] When training a neural network model, multiple sample image pairs from real-world scenes can be acquired. These sample image pairs include sample visible light images and sample infrared images; each sample image pair is taken of the same scene. Based on the features of the sample visible light images, the features of the fused sample image are determined. Based on the features of the fused sample image and the features of the sample infrared image (which was acquired), the neural network model is... Conduct training.
[0106] For example, and The difference loss is as follows: ; where L can be a regression loss function such as MSE or smooth_l1.
[0107] The sample infrared image can be captured by an infrared camera. Based on the different operating wavelengths of infrared cameras, they can be divided into short-wave infrared cameras, mid-wave infrared cameras, and long-wave infrared cameras. If the sample infrared image is obtained by a short-wave infrared camera, then the trained neural network model... It can output short-wave infrared images. Similarly, if the sample infrared image is obtained by a long-wave infrared camera, then the trained neural network model... It can output long-wave infrared images; if the sample infrared image is obtained by a mid-wave infrared camera, then the trained neural network model... It can output mid-wave infrared images
[0108] In one possible implementation, in step 104: the low-frequency spatial features are... and the second feature Before performing fusion and determining the fused image features F, the following process can also be performed: Step 41: Based on low-frequency spatial features Determine the low-frequency spatial feature mask Step 42: Use the low-frequency spatial feature mask. Regarding the second feature The image is processed to obtain a third feature, which represents the high-frequency texture of areas with high transparency in the image. Therefore, when determining the fused image features, the low-frequency spatial feature and the third feature can be fused to determine the fused image features.
[0109] The following provides a detailed description of step 41: Based on the aforementioned low-frequency spatial characteristics When determining the low-frequency spatial feature mask, it can be based on the low-frequency spatial features. Normalization is performed to obtain a low-frequency spatial feature mask. For example, based on the following formula: Determine the low-frequency spatial feature mask.
[0110] Furthermore, a neural network model can be used first. Low-frequency spatial characteristics After processing, normalization is performed to obtain the low-frequency spatial feature mask. For example, based on the following formula: Determine the low-frequency spatial feature mask.
[0111] The following provides a detailed description of step 42: using the aforementioned low-frequency spatial feature mask. Regarding the second feature The process is performed to obtain the third feature. For example, the third feature is: .
[0112] The low-frequency spatial features When fusing with the third feature to determine the fused image feature F, for example, based on the following formula (3): F Determine the features F of the fused image.
[0113] By processing the second feature using a low-frequency feature mask, the enhancement process can be constrained to the outline of a specific object, which is more in line with the visual effect.
[0114] Furthermore, a neural network model can be used first. Low-frequency spatial characteristics After removing redundant information, the feature is fused with the third feature to determine the fused image feature F. For example, based on the following formula (4): F Determine the fused image features F.
[0115] In one possible implementation, mid-frequency spatial features in visible light images can also be determined. ; Regarding the mid-frequency spatial characteristics Perform weakening processing and / or adopt the first feature A fourth feature is obtained by performing a mask-like process. The fourth feature is used to represent the mid-frequency texture of areas with high transparency in a visible light image.
[0116] Introducing mid-frequency features can make the visual effect more natural. Mid-frequency signals can represent rough edges such as object outlines in visible light images. By weakening the mid-frequency spatial features, edge features under low-brightness conditions similar to infrared can be obtained.
[0117] For example, regarding the mid-frequency spatial characteristics The weakening process is as follows: ,in, A value greater than or equal to 2.
[0118] For example, regarding the mid-frequency spatial characteristics The first feature P is used for masking as follows: .
[0119] For example, based on the following formula: Determine the fourth feature .
[0120] Furthermore, a neural network model can be used first. Mid-frequency spatial characteristics After removing redundant information, a weakening process is performed, and the first feature is then applied. A masking process is performed to obtain the fourth feature. For example, based on the following formula: , thus determining the fourth feature.
[0121] Thus, when determining the fused image feature F, specifically, the low-frequency spatial features can be used. The second feature and the fourth feature Perform fusion to determine the fused image features F.
[0122] In the context of the low-frequency spatial characteristics The second feature and the fourth feature When performing fusion and determining the fused image features F, for example, based on the following formula (5): F The fused image feature F is determined. Formula (5) is based on Formula (1) with the addition of a fourth feature. .
[0123] Furthermore, a neural network model can be used first. Low-frequency spatial characteristics After removing redundant information, then the second feature and the fourth feature The fusion process is performed to determine the features of the fused image. For example, based on the following formula (6): F The features of the fused image are determined. Formula (6) is based on Formula (2) with the addition of a fourth feature. .
[0124] Furthermore, for example, based on the following formula (7): F The features of the fused image are determined. Formula (7) is based on Formula (3) with the addition of a fourth feature. .
[0125] Furthermore, for example, based on the following formula (8): F The features of the fused image are determined. Formula (8) is based on Formula (4) with the addition of a fourth feature. .
[0126] Furthermore, in the context of the low-frequency spatial features... The second feature and the fourth feature Before performing fusion and determining the fused image features F, the low-frequency spatial feature mask can also be used as a basis. Regarding the fourth feature The process yields a fifth feature, which represents the mid-frequency texture of highly transparent regions in a visible light image. For example, the fifth feature is... .
[0127] Based on this, the following methods can be used to determine the features of the fused image:
[0128] For example, based on the following formula (9): F The fused image feature F is determined. Formula (9) is based on Formula (1) with the addition of a fifth feature. .
[0129] For example, based on the following formula (10): F The fused image feature F is determined. Formula (10) is based on Formula (2) with the addition of a fifth feature. .
[0130] For example, based on the following formula (11): F The fused image feature F is determined. Formula (11) is based on Formula (3) with the addition of a fifth feature. .
[0131] For example, based on the following formula (12): F The fused image feature F is determined. Formula (12) is based on Formula (4) with the addition of a fifth feature. .
[0132] The following describes step 101: Determining the low-frequency spatial features corresponding to the visible light image. and high-frequency spatial characteristics The process:
[0133] Step 11: Obtain the frequency domain features in the visible light image.
[0134] For example, methods such as Discrete Cosine Transform (DCT), Discrete Wavelet Transform (DWT), and Discrete Fourier Transform (DFT) can be used to obtain frequency domain features in visible light images.
[0135] Visible light images can be in RGB format, or alternatively, they can be converted to RGB format to obtain the frequency domain features of the three colors RGB.
[0136] Step 12: Determine the high-frequency region and high-frequency region in the visible light image according to the pre-set proportion of low-frequency region and high-frequency region in the visible light image.
[0137] The high-frequency region can be specified to be no less than the low-frequency region. For example, the ratio can be 3:2, 1:1, or 4:3, etc.
[0138] For example, the size of a visible light image is 100. Each pixel in 10,000 pixels has its own frequency. Based on this ratio, the number of high-frequency pixels and low-frequency pixels in the 10,000 pixels can be determined. For example, when the ratio is 1:1, there are 5,000 high-frequency pixels and 5,000 low-frequency pixels. Furthermore, the positions of the high-frequency pixels and low-frequency pixels in the visible light image can be determined.
[0139] Step 13: Determine the spatial characteristics of the positions of high-frequency pixels in the visible light image, i.e., high-frequency spatial characteristics; determine the spatial characteristics of the positions of low-frequency pixels in the visible light image, i.e., low-frequency spatial characteristics.
[0140] In one example, a mask method can be used to retain only the feature values of the target frequency (e.g., low or high frequency) and set the feature values of other frequencies to 0. Then, the inverse transform of the corresponding frequency domain method can be used to determine the spatial domain features of the target frequency.
[0141] Alternatively, step 12 can be replaced by: determining the high-frequency, mid-frequency, and high-frequency regions in the visible light image based on a pre-defined ratio of low-frequency, mid-frequency, and high-frequency regions in the visible light image. For example, the ratio could be 4:3:3, 3:3:3, etc. And determining the spatial characteristics of the mid-frequency pixels' positions in the visible light image, i.e., the mid-frequency spatial characteristics.
[0142] This process yields nine spatial features of three different frequencies across the RGB channels. Based on these features, they can be divided into three groups: a high-frequency spatial feature group, a low-frequency spatial feature group, and a mid-frequency spatial feature group. Each group contains three spatial features corresponding to the three channels. These three spatial features in a group can be stitched together to obtain the high-frequency, low-frequency, and mid-frequency spatial features of the visible light image.
[0143] Combination Figure 2 As shown, the process of generating an infrared image is introduced:
[0144] The high-frequency, low-frequency, and mid-frequency spatial features of each channel in an RGB visible light image are determined. The high-frequency spatial features of the three channels are then stitched together to obtain the high-frequency spatial features of the visible light image. The mid-frequency spatial features of the three channels are then stitched together to obtain the mid-frequency spatial features of the visible light image. Finally, the low-frequency spatial features of the three channels are then stitched together to obtain the low-frequency spatial features of the visible light image.
[0145] Based on the low-frequency spatial features and the high-frequency spatial features, the first feature P of the visible light image is determined.
[0146] The high-frequency spatial features are enhanced, and the first feature is used for masking to obtain the second feature.
[0147] The mid-frequency spatial domain feature is weakened, and the first feature is used for mask-like processing to obtain the fourth feature.
[0148] Based on the low-frequency spatial features, a low-frequency spatial feature mask is determined.
[0149] The second feature is processed using the low-frequency spatial feature mask to obtain the third feature.
[0150] The fourth feature is processed using the low-frequency spatial feature mask to obtain the fifth feature.
[0151] The low-frequency spatial features, the third feature, and the fifth feature are fused to determine the fused image features.
[0152] Based on the same technical concept, this application also provides an infrared image generation apparatus. Figure 3 A schematic diagram of an infrared image generation device is shown, including:
[0153] Spatial feature module 31 is used to determine the low-frequency and high-frequency spatial features corresponding to the visible light image;
[0154] Transparency module 32 is used to determine a first feature of the visible light image based on the low-frequency spatial features and the high-frequency spatial features, wherein the first feature is used to represent the transparency of different regions in the visible light image; enhance the high-frequency spatial features and perform mask-like processing using the first feature to obtain a second feature, wherein the second feature is used to represent the high-frequency texture of regions with high transparency in the visible light image.
[0155] The fusion module 33 is used to fuse the low-frequency spatial features and the second feature to determine the fused image features;
[0156] Infrared module 34 is used to input the fused image features into a pre-trained neural network model to obtain an infrared image corresponding to the visible light image.
[0157] One possible implementation also includes:
[0158] Mask module 35 is used to determine a low-frequency spatial feature mask based on the low-frequency spatial feature; and to process the second feature using the low-frequency spatial feature mask to obtain a third feature, wherein the third feature is used to represent the high-frequency texture of the region with high transparency in the visible light image.
[0159] The fusion module 33 is specifically used to fuse the low-frequency spatial features and the third feature to determine the fused image features.
[0160] In one possible implementation, the spatial domain feature module 31 is further used to determine the mid-frequency spatial domain features corresponding to the visible light image;
[0161] The transparency module 32 is also used to weaken the mid-frequency spatial features and perform mask-like processing using the first features to obtain a fourth feature, which is used to represent the mid-frequency texture of the region with high transparency in the visible light image.
[0162] The fusion module 33 is specifically used to fuse the low-frequency spatial features, the second feature, and the fourth feature to determine the fused image features.
[0163] One possible implementation also includes:
[0164] Mask module 35 is used to determine a low-frequency spatial feature mask based on the low-frequency spatial feature; and to process the fourth feature using the low-frequency spatial feature mask to obtain a fifth feature, wherein the fifth feature is used to represent the high-frequency texture of the region with high transparency in the visible light image.
[0165] The fusion module 33 is specifically used to fuse the low-frequency spatial features, the second feature, and the fifth feature to determine the fused image features.
[0166] In one possible implementation, the spatial domain feature module 31 is further used to determine the mid-frequency spatial domain features corresponding to the visible light image;
[0167] The transparency module 32 is also used to weaken the mid-frequency spatial features and perform mask-like processing using the first features to obtain a fourth feature, which is used to represent the mid-frequency texture of the region with high transparency in the visible light image.
[0168] The fusion module 33 is specifically used to fuse the low-frequency spatial features, the third feature, and the fourth feature to determine the fused image features.
[0169] One possible implementation also includes:
[0170] Mask module 35 is used to process the fourth feature using the low-frequency spatial feature mask to obtain a fifth feature, the fifth feature being used to represent the mid-frequency texture of the region with high transparency in the visible light image;
[0171] The fusion module 33 is specifically used to fuse the low-frequency spatial features, the third feature, and the fifth feature to determine the fused image features.
[0172] Based on the same technical concept, this application also provides an electronic device. Figure 4 A schematic diagram of an electronic device structure is shown, such as Figure 4As shown, it includes: processor 41, and optionally, it also includes: communication interface 42, memory 43 and communication bus 44, wherein the processor 41, communication interface 42 and memory 43 communicate with each other through communication bus 44;
[0173] The memory 43 stores a computer program, which, when executed by the processor 41, causes the processor 41 to complete the steps of the infrared image generation method described above.
[0174] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0175] Communication interface 42 is used for communication between the above-mentioned electronic device and other devices.
[0176] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0177] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0178] Based on the same technical concept and the above embodiments, this application provides a computer-readable storage medium storing a computer program executable by an electronic device, wherein computer-executable instructions are used to cause a computer to perform the steps of the above-described infrared image generation method.
[0179] The aforementioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in an electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), optical storage such as CDs, DVDs, BDs, HVDs, etc., and semiconductor storage such as ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs), etc.
[0180] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0181] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0182] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0183] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0184] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method for generating an infrared image, characterized in that, include: Determine the low-frequency and high-frequency spatial features corresponding to the visible light image; Based on the product of the low-frequency spatial features and the high-frequency spatial features, a first feature of the visible light image is determined, which is used to represent the transparency of different regions in the visible light image. The high-frequency spatial features are enhanced, and the product of the first feature and the enhanced high-frequency spatial features is used to obtain the second feature. The second feature is used to represent the high-frequency texture of the region with high transparency in the visible light image. The low-frequency spatial features and the second feature are fused to determine the fused image features; The fused image features are input into a pre-trained neural network model to obtain an infrared image corresponding to the visible light image. The neural network model is used to generate an infrared image based on the input image.
2. The method as described in claim 1, characterized in that, Before fusing the low-frequency spatial features and the second feature to determine the fused image features, the method further includes: Based on the aforementioned low-frequency spatial features, a low-frequency spatial feature mask is determined; The second feature is processed using the low-frequency spatial feature mask to obtain a third feature, which is used to represent the high-frequency texture of the highly transparent region in the visible light image. The low-frequency spatial features and the second feature are fused to determine the fused image features, including: The low-frequency spatial features and the third feature are fused to determine the fused image features.
3. The method as described in claim 1, characterized in that, Before fusing the low-frequency spatial features and the second feature to determine the fused image features, the method further includes: Determine the mid-frequency spatial features corresponding to the visible light image; The mid-frequency spatial domain feature is weakened, and the product of the first feature and the weakened mid-frequency spatial domain feature is used to obtain a fourth feature, which is used to represent the mid-frequency texture of the region with high transparency in the visible light image. The low-frequency spatial features and the second feature are fused to determine the fused image features, including: The low-frequency spatial features, the second feature, and the fourth feature are fused to determine the fused image features.
4. The method as described in claim 3, characterized in that, Before fusing the low-frequency spatial features, the second feature, and the fourth feature to determine the fused image features, the method further includes: Based on the aforementioned low-frequency spatial features, a low-frequency spatial feature mask is determined; The fourth feature is processed using the low-frequency spatial feature mask to obtain the fifth feature, which is used to represent the high-frequency texture of the highly transparent region in the visible light image. The low-frequency spatial features, the second feature, and the fourth feature are fused to determine the fused image features, including: The low-frequency spatial features, the second feature, and the fifth feature are fused to determine the fused image features.
5. The method as described in claim 2, characterized in that, Before fusing the low-frequency spatial features and the third feature to determine the fused image features, the process further includes: Determine the mid-frequency spatial features corresponding to the visible light image; The mid-frequency spatial domain feature is weakened, and the product of the first feature and the weakened mid-frequency spatial domain feature is used to obtain a fourth feature, which is used to represent the mid-frequency texture of the region with high transparency in the visible light image. The low-frequency spatial features and the third feature are fused to determine the fused image features, including: The low-frequency spatial features, the third feature, and the fourth feature are fused to determine the fused image features.
6. The method as described in claim 5, characterized in that, Before fusing the low-frequency spatial features, the third feature, and the fourth feature to determine the fused image features, the process further includes: The fourth feature is processed using the low-frequency spatial feature mask to obtain the fifth feature, which is used to represent the mid-frequency texture of the highly transparent region in the visible light image. The low-frequency spatial features, the third feature, and the fourth feature are fused to determine the fused image features, including: The low-frequency spatial features, the third feature, and the fifth feature are fused to determine the fused image features.
7. The method as described in claim 1, characterized in that, The infrared image is a shortwave infrared image.
8. An infrared image generation apparatus, characterized in that, include: The spatial domain feature module is used to determine the low-frequency and high-frequency spatial domain features corresponding to the visible light image; A transparency module is used to determine a first feature of the visible light image based on the product of the low-frequency spatial features and the high-frequency spatial features. The first feature is used to represent the transparency of different regions in the visible light image. The high-frequency spatial features are enhanced, and a second feature is obtained by multiplying the first feature and the enhanced high-frequency spatial features. The second feature is used to represent the high-frequency texture of regions with high transparency in the visible light image. The fusion module is used to fuse the low-frequency spatial features and the second feature to determine the fused image features; An infrared module is used to input the fused image features into a pre-trained neural network model to obtain an infrared image corresponding to the visible light image. The neural network model is used to generate an infrared image based on the input image.
9. An electronic device, characterized in that, include: Processor and memory; The memory is used to store computer programs or instructions; The processor is configured to execute some or all of the computer programs or instructions in the memory, and when the some or all of the computer programs or instructions are executed, to implement the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, the computer program including instructions for implementing the method of any one of claims 1-7.