Color transmission method and system based on visible light image and infrared image
Through the color transfer method of blocking the image and type judgment, the color transfer problem when the image scene is complex in the prior art is solved, and the clarity of the image and the authenticity of the color are improved, especially suitable for the color processing of natural scenery. .
Patent Information
- Application Number
- CN202411313183.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2025-01-03
AI Technical Summary
When processing images, when the prior art, when the scene content is complex, color transmission is prone to color strings or scene color turbidity, resulting in a decrease in image clarity and affecting observation speed and accuracy.
Using a color transfer method based on visible light images and infrared images, a dictionary library is constructed by obtaining reference images of different scene types, and image blocking is performed, and color transfer is performed according to the type of visible light image blocks to generate color fusion images to reduce the mutual connection between scenes. Interference and improve the accuracy of color transmission.
It improves the accuracy of local colors in the image, makes the colors of natural scenery more realistic, avoids mutual interference and color errors in the image scene, and enhances the clarity and authenticity of the image.
Smart Images
Figure CN119251126A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a color transfer method and system based on visible light images and infrared images. Background Art
[0002] The color processing of infrared and visible light image fusion belongs to pseudo color processing. Color transfer is to transfer the color distribution of the reference image to the fused grayscale image to achieve the purpose of highlighting the target and improving the contrast. At present, the fusion color processing is mainly through the global statistics and calculation of the reference image and the image to be processed; for example, the TNO method is to calculate the corresponding value of each pixel after processing by counting the global mean and variance of each channel of the RGB or YUV image, and then calculate the corresponding value of each pixel through the transfer formula; the Welsh algorithm is also for global image processing.
[0003] The processing idea of the existing technology is mainly through calculation of the global color, which also determines that the processing method of the existing technology is better for images with a single color; when the scene content in the image is complex, the use of the existing technology for color transmission will cause color bleeding or turbid scene colors, which will cause the image clarity to decrease. The decrease in image clarity may lead to a reduction in the speed and accuracy of observation. Summary of the invention
[0004] In order to help solve the problem that when the scene content in the image is complex, the use of existing technology for color transfer will cause color bleeding or scene color turbidity, thereby causing a decrease in image clarity. The decrease in image clarity may lead to a reduction in observation speed and accuracy. The present application provides a color transfer method and system based on visible light images and infrared images.
[0005] In a first aspect, the present application provides a color transfer method based on visible light images and infrared images, which adopts the following technical solution: the method comprises:
[0006] Acquire reference images corresponding to different scene types, and construct a dictionary library according to the reference images, wherein the dictionary library includes scene dictionaries corresponding to different scene types;
[0007] Acquire an original visible light image and an original infrared image, perform grayscale multi-scale fusion on the original visible light image and the original infrared image, and generate a grayscale fused image;
[0008] The original visible light image and the grayscale fused image are processed by block division according to a preset image block size to generate a plurality of visible light image blocks and a plurality of grayscale fused image blocks, wherein the visible light image blocks correspond to the grayscale fused image blocks and the number of the blocks is the same;
[0009] Determine the types of the visible light image blocks according to the visible light image blocks and the dictionary library, and transfer the color of the reference image corresponding to the type of the visible light image block to the grayscale fused image block by using a color transfer algorithm to generate a plurality of color fused image blocks;
[0010] A plurality of the color fusion image blocks are spliced and fused to generate a color image.
[0011] In a specific implementation manner, constructing a dictionary library according to the reference image includes:
[0012] Performing image block processing on the reference images corresponding to all scene types according to a preset image block size to generate a plurality of reference image blocks;
[0013] Extracting feature information of a number of reference image blocks to generate a number of block feature vectors;
[0014] Combining a number of the block feature vectors to generate a reference image feature vector, and generating an incomplete dictionary corresponding to all scene types according to the reference image feature vector;
[0015] Using the K-SVD algorithm to thin out the incomplete dictionary corresponding to each scene type, so as to generate scene dictionaries corresponding to different scene types;
[0016] The scene dictionaries corresponding to all scene types are combined to form the dictionary library.
[0017] In a specific implementation manner, determining the types of the plurality of visible light image blocks according to the visible light image blocks and the dictionary library comprises:
[0018] Extracting feature information of the visible light image block and generating a number of visible light image block feature vectors;
[0019] Calculating the sparse coefficient of the current visible light image block under each of the scene dictionaries using an energy formula according to the visible light image block feature vector of the current visible light image block;
[0020] The scene type of the scene dictionary corresponding to the calculated sparse coefficient with the largest absolute value is set as the type of the current visible light image block;
[0021] The types of all the visible light image blocks are calculated and determined in sequence.
[0022] In a specific embodiment, the energy formula includes:
[0023]
[0024] Among them, β represents the sparse coefficient, λ represents the constant, and D Grepresents the reference image feature vector, g i Represents the visible light image block feature vector.
[0025] In a specific implementation manner, after the color image is generated by splicing and fusing a plurality of the color fused image blocks, the method further includes:
[0026] A filtering algorithm is used to filter the fused color image to generate a color clear image.
[0027] In a second aspect, the present application provides a color transfer device, which adopts the following technical solution: the device comprises:
[0028] A dictionary library construction module, used to obtain reference images corresponding to different scene types, and to construct a dictionary library according to the reference images, wherein the dictionary library includes scene dictionaries corresponding to different scene types;
[0029] A grayscale image generation module is used to obtain an original visible light image and an original infrared image, perform grayscale multi-scale fusion on the original visible light image and the original infrared image, and generate a grayscale fused image;
[0030] An image segmentation module, used for performing block processing on the original visible light image and the grayscale fused image according to a preset image block size to generate a plurality of visible light image blocks and a plurality of grayscale fused image blocks, wherein the visible light image blocks correspond to the grayscale fused image blocks and the number of the blocks is the same;
[0031] A color transfer module, used for determining the types of the visible light image blocks according to the visible light image blocks and the dictionary library, and transferring the color of the reference image corresponding to the type of the visible light image block to the grayscale fused image block by using a color transfer algorithm, so as to generate a plurality of color fused image blocks;
[0032] The color image generation module is used to splice and fuse a plurality of the color fusion image blocks to generate a color image.
[0033] In a specific implementation scheme, the dictionary library construction module is specifically used to:
[0034] Performing image block processing on the reference images corresponding to all scene types according to a preset image block size to generate a plurality of reference image blocks;
[0035] Extracting feature information of a number of reference image blocks to generate a number of block feature vectors;
[0036] Combining a number of the block feature vectors to generate a reference image feature vector, and generating an incomplete dictionary corresponding to all scene types according to the reference image feature vector;
[0037] Using the K-SVD algorithm to thin out the incomplete dictionary corresponding to each scene type, so as to generate scene dictionaries corresponding to different scene types;
[0038] The scene dictionaries corresponding to all scene types are combined to form the dictionary library.
[0039] In a specific embodiment, the color transfer module is specifically used to:
[0040] Extracting feature information of the visible light image block and generating a number of visible light image block feature vectors;
[0041] Calculating the sparse coefficient of the current visible light image block under each of the scene dictionaries using an energy formula according to the visible light image block feature vector of the current visible light image block;
[0042] The scene type of the scene dictionary corresponding to the calculated sparse coefficient with the largest absolute value is set as the type of the current visible light image block;
[0043] The types of all the visible light image blocks are calculated and determined in sequence.
[0044] In a third aspect, the present application provides a computer device, which adopts the following technical solution: it includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute any of the above-mentioned color transfer methods.
[0045] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute any of the above-mentioned color transfer methods.
[0046] In summary, this application has the following beneficial technical effects:
[0047] By processing the image in blocks, determining the type of each image block, and then transferring colors according to the type of the image block, the mutual interference of the scenes in the scene can be eliminated, and at the same time, the boundary between different scenes can be determined, so that the assigned color can be close to the real color of the natural scene, and the error of color assignment can be reduced. In the prior art, based on global calculation and color transfer of the global image, the present application scheme realizes local color transfer of the image by means of image blocks, so as to improve the accuracy of color transfer, make the local color closer to the real scene, and especially make the color of the natural scene more real. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is a flow chart of a color transfer method in an embodiment of the present application;
[0049] Figure 2is a schematic diagram of an original visible light image in an embodiment of the present application;
[0050] Figure 3 is a schematic diagram of an original infrared image in an embodiment of the present application;
[0051] Figure 4 is a color image after FPGA global color processing in an embodiment of the present application;
[0052] Figure 5 is a color image adjusted by FPGA local color processing in an embodiment of the present application;
[0053] Figure 6 is a schematic diagram of a color transfer device in an embodiment of the present application;
[0054] Figure 7 It is a schematic diagram used to embody a computer device in an embodiment of the present application.
[0055] Attached figure numerals: 601, dictionary library construction module; 602, grayscale image generation module; 603, image segmentation module; 604, color transfer module; 605, color image generation module. DETAILED DESCRIPTION
[0056] The following combination Figure 1-Figure 7 This application is described in further detail.
[0057] The embodiment of the present application discloses a color transfer method based on visible light images and infrared images. This method can be used to determine the type of image blocks, and perform color transfer of the corresponding reference image according to the type of image blocks, so that local colors are closer to real scenes, especially making the colors of natural scenes more realistic, thereby avoiding the problem of color bleeding or scene color turbidity when the image scene content is complex.
[0058] The color processing of infrared and visible light image fusion belongs to pseudo color processing. Color transfer is to transfer the color distribution of the reference image to the fused grayscale image to achieve the purpose of highlighting the target and improving the contrast. The current color processing methods include: linear transfer in RGB, Lab, YUV and other color spaces, sparse coding color reconstruction, direct color channel mapping method, etc.
[0059] Commonly used color transfer algorithms include TNO color transfer algorithm and Welsh color transfer algorithm. The linear transfer of color for image fusion was mainly proposed by TNO; after years of development, multiple transfer, multi-scale transfer and other methods have also evolved. This method can facilitate the real-time implementation.
[0060] Welsh color migration method: First, convert the reference image and grayscale image from RGB space to lαβ color space respectively. Second, perform brightness remapping on the reference image according to the brightness and standard deviation of the grayscale image. Since the histogram values of the reference image and the grayscale image are not necessarily in the same range, a large error will occur if a reference image with a very low brightness value is compared with a histogram of a target image with a very high brightness value. Therefore, it is necessary to perform brightness remapping on the reference image. The remapping method can be expressed as: L = (nl' / nl)*(l–ml)+ml'; where l is the data of the l channel of the source image, L is the value of the l channel of the new source image after the transformation, ml and ml' are the means of the l channels of the source image and the colored image, respectively, and nl and nl' represent their l channel standard deviations. Third, randomly select a batch of sample points from the reference image, and use the linear combination of the brightness of the pixel and the standard deviation of the brightness in the neighborhood as the weight. The calculation formula can be expressed as: W = l / 2 + σ / 2; where w is the weight, l is the brightness of the pixel, and σ is the standard deviation of the brightness in a neighborhood around the pixel; Regarding the size of the neighborhood, Welsh pointed out that it is generally 5×5, and a larger neighborhood will have better results for individual images. Fourth, scan the grayscale image row by row, and for each pixel, calculate its weight by the formula, and find the closest sample point among the sample points of the reference image, that is, the best matching point, and assign the value of the color channel of the matching point to the corresponding pixel of the grayscale image, thereby realizing color transfer. Fifth, convert the reference image and grayscale image from lαβ space to RGB space.
[0061] From the above TNO algorithm and Welsh algorithm, we can see that the current fusion color processing is mainly through global statistics and calculations of the reference image and the image to be processed; for example, the TNO method is to calculate the global mean and variance of each channel of the RGB or YUV image, and then calculate the corresponding value of each pixel after processing through the transfer formula; the Welsh algorithm is also for global image processing.
[0062] The processing idea of the prior art through the calculation of global color determines that the processing method of the prior art has a better effect on images with a single color; when the scene content in the image is complex, the use of the prior art for color transfer will cause color bleeding or turbid scene color, which will cause the image clarity to decrease. The decrease in image clarity may lead to a decrease in the speed and accuracy of observation; in addition, due to global statistical calculations, the processed color of the scene depends on the result of global color balance, resulting in large color errors in the coloring process, which cannot maintain authenticity and is unnatural, and it is difficult to restore the natural color of the original scene. In order to help avoid the problems of color bleeding and turbid scene colors, as well as large color errors in the coloring process that are unrealistic and unnatural, the present application provides a color transfer method based on visible light images and infrared images.
[0063] Reference Figure 1 , the method comprises the following steps:
[0064] S10, obtaining reference images corresponding to different scene types, and constructing a dictionary library according to the reference images, wherein the dictionary library includes scene dictionaries corresponding to different scene types.
[0065] Specifically, in order to improve the accuracy of color migration, the accuracy of color migration can be improved by judging the image type before color migration. At the same time, the local color of the image after color transfer can also be closer to the real scene, making the color of the scene more realistic. Scenery types can be divided into natural background scenes and artificial background scenes. Natural background scenes can include the sky, plants, bare land, water surface, desert, shallow grass, the junction of heaven and earth, the junction of sea and sky, etc.; artificial background scenes can include asphalt roads, buildings, etc., and scene dictionaries are constructed for different types of scenes. The scene dictionaries of all scene types are combined to form a dictionary library; wherein, the scene types used to construct the dictionary library can be selected by the user according to actual needs. Specifically, the construction of scene dictionaries corresponding to different types of scenes is based on reference images in advance. The reference image can be understood as an image of the standard color of the scene. Through color transfer, the standard color of the scene is transferred to the grayscale image to achieve coloring of the grayscale image.
[0066] S20, acquiring an original visible light image and an original infrared image, performing grayscale multi-scale fusion on the original visible light image and the original infrared image, and generating a grayscale fused image.
[0067] Specifically, obtain the original visible light image, refer to Figure 2 , and the original infrared image reference Figure 3 ; Among them, the original visible light image and the original infrared image can be understood as the visible light image and infrared image to be processed, and the original visible light image and the original infrared image are two images taken at the same time and at the same position; because the visible light image has richer texture but is easily affected by environmental factors, while the infrared image is less affected by environmental factors, it can still accurately capture the characteristic information of significant targets even in complex environments such as darkness and dense fog, but the infrared image has relatively less texture information. Therefore, by fusing the original visible light image and the original infrared image, an image with richer image information and higher definition can be obtained, so that the targets in the image can be identified and detected later. Since the color processing of the fusion of infrared and visible light images belongs to pseudo-color processing, the original infrared image and the original visible light image are subjected to grayscale multi-scale fusion processing to obtain a grayscale fused image, and then the grayscale fused image is assigned color to obtain a color image with higher definition.
[0068] S30, performing block processing on the original visible light image and the grayscale fused image according to a preset image block size to generate a plurality of visible light image blocks and a plurality of grayscale fused image blocks, wherein the visible light image blocks correspond to the grayscale fused image blocks and the number of the visible light image blocks is the same.
[0069] Specifically, in the embodiment of the present application, color transfer is performed in the form of image blocks, and the original visible light image and the grayscale fused image are divided into blocks according to a preset image block size to generate a number of visible light image blocks and a number of grayscale fused image blocks, wherein the visible light image blocks correspond to the grayscale fused image blocks and the number is the same. It can also be understood that the visible light image and the grayscale fused image are divided into blocks in the same manner and according to the same image block size, so that the visible light image blocks and the grayscale fused image blocks can correspond one to one, which is convenient for subsequent color transfer of the grayscale image.
[0070] It should be noted that the preset image block size and image segmentation method can be selected and set by the user according to actual needs. In the embodiment of the present application, the image is segmented by a sliding window. For example, assuming that the preset image block size is set to 8*8, the offset step of each small block is 4 during the sliding segmentation process, and there is an overlapping area between each small block, so as to avoid missing information such as image features and details between image blocks as much as possible, and ensure the richness of the obtained image information. In actual applications, users can make specific settings based on the computing resources of the actual processing equipment, the computing amount of the task, etc.
[0071] S40, determining types of several visible light image blocks according to the visible light image blocks and the dictionary library, and transferring the color of the reference image corresponding to the visible light image block type to the grayscale fused image block using a color transfer algorithm to generate several color fused image blocks.
[0072] Specifically, the types of several visible light image blocks are determined based on the visible light image blocks and the dictionary library. Since the texture information in the visible light image is relatively rich, the use of visible light images for type determination can improve the accuracy of type determination. The dictionary library contains a relatively comprehensive range of scene types. The visible light image blocks of the type to be determined are compared with the dictionary, so that the types of the visible light image blocks can be determined. After the types of all visible light images are determined, the colors of the reference images corresponding to the types of the visible light image blocks are transferred to the grayscale fusion image blocks using a color transfer algorithm to generate several color fusion image blocks. Specifically, the color transfer algorithm used can be the TNO color transfer algorithm. At this time, the color transfer only transfers the color, and the image grayscale does not need to be transferred.
[0073] S50, splicing and fusing a plurality of color fusion image blocks to generate a color image.
[0074] Specifically, the generated color fusion image blocks are stitched or fused together to generate a final color image, thereby completing the color assignment process of the grayscale image.
[0075] In the present application, the image is processed in blocks, the type of each image block is determined, and then color transfer is performed according to the type of the image block, which can eliminate the mutual interference of the scenes in the scene, and at the same time, the boundary of different scenes can be determined, so that the assigned color can be close to the real color of the natural scene, and the error of color assignment can be reduced. In the prior art, based on global calculation and color transfer of the global image, the present application realizes local color transfer of the image by image blocks compared to the prior art, so as to improve the accuracy of color transfer, make the local color closer to the real scene, and especially make the color of the natural scene more real.
[0076] In one embodiment, the step of constructing a dictionary library according to the reference image may be specifically performed as follows:
[0077] In the embodiment of the present application, the process of constructing the dictionary library can be implemented by using a host computer. Specifically, first, the reference images corresponding to all scene types are divided into blocks according to the preset image block size to generate a number of reference image blocks. The host computer is relatively rich in computing resources and can process a large amount of data. Therefore, in the embodiment of the present application, a scene dictionary is constructed for 10 scene types, including the sky, plants, bare land, water surface, desert, shallow grass, the junction of heaven and earth, the junction of sea and sky, asphalt road surface, and buildings; the reference images used to construct the dictionary can be mostly established using daytime pictures. In order to ensure the accuracy of the constructed dictionary, multiple reference images can be used to establish it. For example, in the embodiment of the present application, the dictionary of each scene can be represented by 4 pictures. The specific number of images can be set by the user according to the computing resources and task requirements of the host computer in actual applications, and there is no restriction here. Among them, for the selection of reference images, the classification of the scene content in the reference image is judged. The size of the scene content in the reference image must be greater than 3 / 4 of the total size of the image. For example, a reference image with a size of 32*32 contains content such as the sky and marine vegetation, and the size of the sky part is greater than 3 / 4 of the total size of the image, then the reference image is classified as a sky type. After selecting and determining different types of reference images, all reference images are divided into small image blocks. The image block size and image segmentation method can be set according to actual needs. In the embodiment of the present application, the image is divided into 8*8 small blocks by sliding segmentation, and the offset step size of each small block is set to 4.
[0078] Each image block contains image feature information. After the image is divided into blocks, the feature information of several reference image blocks is extracted. The feature information of the image block includes two parts: brightness and gradient. The feature information of each reference image block is a column vector, and the block feature vector consistent with the number of image blocks is generated. When the image is divided into 8*8 blocks by sliding segmentation, the number of elements in each column vector is 192.
[0079] Afterwards, several block feature vectors are combined to generate a reference image feature vector, and an incomplete dictionary corresponding to all scene types is generated based on the reference image feature vector. It can also be understood that the reference image feature vectors corresponding to all reference images are combined together to form a joint incomplete dictionary, and the incomplete dictionary is Gaussian normalized to ensure consistency between data, improve data processing efficiency and model stability.
[0080] Finally, the K-SVD algorithm is used to sparse the incomplete dictionary corresponding to each scene type to generate scene dictionaries corresponding to different scene types. The generated scene dictionary is a sparse dictionary. The three parts of information contained in the dictionary are represented as D L (brightness information), D F (gradient information), D C (color information), when the image is divided into 8*8 small blocks by sliding segmentation, the number of elements in each column vector is 192, and after sparseness, a 49*256 dictionary matrix is formed; then the scene dictionaries corresponding to all scene types are combined to construct a generated dictionary library.
[0081] In the present application, 10 types of scene types are selected. The more types of scenes there are, the more accurate the division is, and the more accurate the final color transfer result is, thereby improving the authenticity of the image color; constructing a scene dictionary and generating a dictionary library, and performing sparse processing on the dictionary can effectively capture the key information in the image, while reducing redundant information, thereby improving processing efficiency.
[0082] In one embodiment, the step of determining the types of the plurality of visible light image blocks according to the visible light image blocks and the dictionary library may be specifically performed as follows:
[0083] In the embodiment of the present application, the host computer can be used to implement the type judgment process. First, since the visible light image has rich texture information and is more suitable for type judgment, the feature information of the visible light image block is extracted to generate several visible light image block feature vectors. The feature extraction and visible light image block feature vector generation methods are consistent with the reference image block feature extraction method. For example, in the embodiment of the present application, the visible light image is also divided into 8*8 small blocks, and the offset step size of each small block is 4. The feature information of each small block includes brightness information and gradient information. The feature information of each small block is set as a column vector, and the number of elements in the column vector is 192. The generated visible light image block feature vector is also Gaussian normalized.
[0084] Afterwards, according to the visible light image block feature vector of the current visible light image block, the energy formula is used to calculate the sparse coefficient of the current visible light image block under each scene dictionary; using the energy formula to calculate the sparse coefficient can also be understood as using the OMP algorithm to solve the sparse coefficient of the current visible light image block under each scene dictionary. Finally, the scene type of the scene dictionary corresponding to the sparse coefficient with the largest absolute value calculated is set as the type of the current visible light image block; the types of all visible light image blocks are calculated and determined in sequence.
[0085] The energy formula can be expressed as:
[0086]
[0087] D G =[D L ,D F ] T ,g i =[l i ,f i ] T
[0088] Among them, β represents the sparse coefficient, λ represents a constant, which is generally taken as 0.5, and D G represents the reference image feature vector, g i represents the feature vector of the visible light image block, l i Represents the brightness information of the visible light image block, f i Represents the visible light image block gradient information.
[0089] In the present application scheme, the visible light image block is sparsely represented in a sparse dictionary, that is, a scene dictionary. The size of the sparse coefficient reflects the strength of the image block feature. By calculating the sparse coefficient of the visible light image block under each sparse dictionary, the scene type of the scene dictionary with the largest absolute value of the sparse coefficient is selected and set as the type of the current visible light image block, thereby improving the accuracy of type judgment, and further improving the accuracy of subsequent color transmission and the authenticity of the final color image.
[0090] In one embodiment, considering that the image blocks used in the present application scheme for color transfer processing may produce a block effect, after a plurality of color fused image blocks are spliced and fused to generate a color image, the following steps may be performed:
[0091] A filtering algorithm is used to filter the fused color image to generate a color clear image. The filtering algorithm can be a median filtering algorithm or a multi-scale method.
[0092] In the present application scheme, considering that the image blocks used in the present application scheme for color transfer processing will produce block effects, and interference may occur between image blocks, the block effects are eliminated by filtering to improve the clarity and quality of the image.
[0093] Figure 1 FIG. 1 is a flow chart of a color transfer method based on a visible light image and an infrared image in one embodiment. Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps may be executed in other orders; and Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0094] Based on the above method, an embodiment of the present application also discloses a color transfer system based on visible light images and infrared images.
[0095] Reference Figure 6 The system includes the following modules:
[0096] A dictionary library construction module 601 is used to obtain reference images corresponding to different scene types and construct a dictionary library according to the reference images, wherein the dictionary library includes scene dictionaries corresponding to different scene types;
[0097] A grayscale image generation module 602 is used to obtain an original visible light image and an original infrared image, perform grayscale multi-scale fusion on the original visible light image and the original infrared image, and generate a grayscale fused image;
[0098] An image segmentation module 603 is used to perform block processing on the original visible light image and the grayscale fused image according to a preset image block size to generate a number of visible light image blocks and a number of grayscale fused image blocks, where the visible light image blocks correspond to the grayscale fused image blocks and the number of the blocks is the same;
[0099] A color transfer module 604 is used to determine the types of several visible light image blocks according to the visible light image blocks and the dictionary library, and transfer the color of the reference image corresponding to the type of the visible light image block to the grayscale fused image block using a color transfer algorithm to generate several color fused image blocks;
[0100] The color image generation module 605 is used to splice and fuse a number of color fusion image blocks to generate a color image.
[0101] In one embodiment, the dictionary library construction module 601 is specifically used to perform image block processing on reference images corresponding to all scene types according to a preset image block size to generate a number of reference image blocks; extract feature information of the reference image blocks to generate a number of block feature vectors; combine the several block feature vectors to generate a reference image feature vector, and generate an incomplete dictionary corresponding to all scene types based on the reference image feature vector; use the K-SVD algorithm to sparse the incomplete dictionary corresponding to each scene type to generate scene dictionaries corresponding to different scene types; and combine the scene dictionaries corresponding to all scene types to generate a dictionary library.
[0102] In one embodiment, the color transfer module 604 is specifically used to extract feature information of a visible light image block and generate a plurality of visible light image block feature vectors; calculate the sparse coefficient of the current visible light image block under each scene dictionary using an energy formula according to the visible light image block feature vector of the current visible light image block; set the scene type of the scene dictionary corresponding to the sparse coefficient having the largest calculated absolute value as the type of the current visible light image block; and sequentially calculate and determine the types of all visible light image blocks.
[0103] In one embodiment, the color transfer module 604 specifically includes:
[0104]
[0105] Among them, β represents the sparse coefficient, λ represents the constant, and D G represents the reference image feature vector, g i Represents the visible light image block feature vector.
[0106] In one embodiment, the color image generation module 605 is further configured to use a filtering algorithm to filter the fused color image to generate a color clear image.
[0107] The color transfer system based on visible light images and infrared images provided in the embodiments of the present application can be applied to the color transfer method based on visible light images and infrared images provided in the above embodiments. For relevant details, refer to the above method embodiments. The implementation principles and technical effects are similar and will not be repeated here.
[0108] It should be noted that: the color transfer system based on visible light images and infrared images provided in the embodiments of the present application only uses the division of the above-mentioned functional modules / functional units as an example when performing color transfer. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the color transfer system based on visible light images and infrared images is divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the color transfer method based on visible light images and infrared images provided in the above method embodiment and the implementation method of the color transfer system based on visible light images and infrared images provided in this embodiment belong to the same concept. The specific implementation process of the color transfer system based on visible light images and infrared images provided in this embodiment is detailed in the above method embodiment, which will not be repeated here.
[0109] The embodiment of the present application also discloses a computer device.
[0110] Specifically, if Figure 7 As shown, the computer device can be a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. The processor and the memory may be connected via a bus or otherwise. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processors (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips.
[0111] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method implementation is realized. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.
[0112] The embodiment of the present application also discloses a computer-readable storage medium.
[0113] Specifically, a computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art will understand that the implementation of all or part of the process in the above-mentioned implementation method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the process of the implementation of the above-mentioned methods. Among them, the storage medium may be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memory.
[0114] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.
Claims
1. A color transfer method based on visible light image and infrared image, characterized in that: The method comprises: Acquire reference images corresponding to different scene types, and construct a dictionary library according to the reference images, wherein the dictionary library includes scene dictionaries corresponding to different scene types; Acquire an original visible light image and an original infrared image, perform grayscale multi-scale fusion on the original visible light image and the original infrared image, and generate a grayscale fused image; The original visible light image and the grayscale fused image are processed by block division according to a preset image block size to generate a plurality of visible light image blocks and a plurality of grayscale fused image blocks, wherein the visible light image blocks correspond to the grayscale fused image blocks and the number of the blocks is the same; Determine the types of the visible light image blocks according to the visible light image blocks and the dictionary library, and transfer the color of the reference image corresponding to the type of the visible light image block to the grayscale fused image block by using a color transfer algorithm to generate a plurality of color fused image blocks; A plurality of the color fusion image blocks are spliced and fused to generate a color image.
2. The method according to claim 1, characterized in that: The constructing a dictionary library according to the reference image comprises: Performing image block processing on the reference images corresponding to all scene types according to a preset image block size to generate a plurality of reference image blocks; Extracting feature information of a number of reference image blocks to generate a number of block feature vectors; Combining a number of the block feature vectors to generate a reference image feature vector, and generating an incomplete dictionary corresponding to all scene types according to the reference image feature vector; Using the K-SVD algorithm to thin out the incomplete dictionary corresponding to each scene type, so as to generate scene dictionaries corresponding to different scene types; The scene dictionaries corresponding to all scene types are combined to form the dictionary library.
3. The method according to claim 2, characterized in that: The determining the types of the plurality of visible light image blocks according to the visible light image blocks and the dictionary library comprises: Extracting feature information of the visible light image block and generating a number of visible light image block feature vectors; Calculating the sparse coefficient of the current visible light image block under each of the scene dictionaries using an energy formula according to the visible light image block feature vector of the current visible light image block; The scene type of the scene dictionary corresponding to the calculated sparse coefficient with the largest absolute value is set as the type of the current visible light image block; The types of all the visible light image blocks are calculated and determined in sequence.
4. The method according to claim 3, characterized in that: The energy formula includes: Where β represents the sparse coefficient, λ represents a constant, D represents the reference image feature vector, g i Represents the visible light image block feature vector.
5. The method according to claim 1, characterized in that: After the color fusion image blocks are spliced and fused to generate a color image, the method further includes: A filtering algorithm is used to filter the fused color image to generate a color clear image.
6. A color transfer system based on visible light images and infrared images, characterized in that: The system comprises: A dictionary library construction module (601) is used to obtain reference images corresponding to different scene types and construct a dictionary library based on the reference images, wherein the dictionary library includes scene dictionaries corresponding to different scene types; A grayscale image generation module (602) is used to obtain an original visible light image and an original infrared image, perform grayscale multi-scale fusion on the original visible light image and the original infrared image, and generate a grayscale fused image; An image segmentation module (603) is used to perform block processing on the original visible light image and the grayscale fused image according to a preset image block size to generate a plurality of visible light image blocks and a plurality of grayscale fused image blocks, wherein the visible light image blocks correspond to the grayscale fused image blocks and the number of the blocks is the same; A color transfer module (604) is used to determine the types of the visible light image blocks according to the visible light image blocks and the dictionary library, and transfer the color of the reference image corresponding to the type of the visible light image block to the grayscale fused image block using a color transfer algorithm to generate a plurality of color fused image blocks; The color image generation module (605) is used to splice and fuse a plurality of the color fusion image blocks to generate a color image.
7. The system according to claim 6, characterized in that: The dictionary library construction module (601) is specifically used for: Performing image block processing on the reference images corresponding to all scene types according to a preset image block size to generate a plurality of reference image blocks; Extracting feature information of a number of reference image blocks to generate a number of block feature vectors; Combining a number of the block feature vectors to generate a reference image feature vector, and generating an incomplete dictionary corresponding to all scene types according to the reference image feature vector; Using the K-SVD algorithm to thin out the incomplete dictionary corresponding to each scene type, so as to generate scene dictionaries corresponding to different scene types; The scene dictionaries corresponding to all scene types are combined to form the dictionary library.
8. The system according to claim 6, characterized in that: The color transfer module (604) is specifically used for: Extracting feature information of the visible light image block and generating a number of visible light image block feature vectors; Calculating the sparse coefficient of the current visible light image block under each of the scene dictionaries using an energy formula according to the visible light image block feature vector of the current visible light image block; The scene type of the scene dictionary corresponding to the calculated sparse coefficient with the largest absolute value is set as the type of the current visible light image block; The types of all the visible light image blocks are calculated and determined in sequence.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes any one of the methods of claims 1 to 5.
10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 5.