Image processing model training method and processing method, system, device and medium

Through the image processing model training method, combined with image color and idempotent operators, the shortcomings of infrared images and visible light images are solved, efficient and accurate image fusion is achieved, and analytical images with rich content are generated.

CN118470465BActive Publication Date: 2024-11-05HARBIN INSTITUTE OF TECHNOLOGY (SHENZHEN) (INSTITUTE OF SCIENCE AND TECHNOLOGY INNOVATION HARBIN INSTITUTE OF TECHNOLOGY SHENZHEN) +1
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410906787.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-05
Estimated Expiration
2044-07-08

AI Technical Summary

Technical Problem

The combination of infrared images and visible light images in the prior art has not been widely recognized in performance and satisfaction, and there is a lack of effective fusion methods and cannot meet daily needs.

Method used

A method of image processing model training is proposed. By inputting visible light images and infrared images of the same picture into multiple hidden layers, calculating multiple loss values ​​and updating model parameters, combining image color, content and idempotent operators, improving fusion efficiency and accuracy.

Benefits of technology

It effectively improves the fusion efficiency and accuracy of infrared images and visible light images, provides stable fusion results, and generates analytical images with rich image content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118470465B_ABST
    Figure CN118470465B_ABST
Patent Text Reader

Abstract

The present invention relates to an image processing model training method and processing method, system, device and medium. The image processing model includes several hidden layers, and the visible light image and infrared image of the same picture are used as the input information of the starting hidden layer of the image processing model to obtain the output information of the starting hidden layer; the output information of the starting hidden layer is used as the input information of the next hidden layer to obtain the output information of the next hidden layer, until the target image output by the last hidden layer is obtained; the first loss value is calculated according to the target image and the visible light image and the infrared image; the second loss value is calculated according to the target image and the visible light image; the third loss value is calculated according to the information output by each hidden layer; the parameters of the image processing model are updated according to the first loss value, the second loss value and the third loss value. The problem that the existing image fusion cannot effectively combine the visible light image and the infrared image is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image melting, and in particular to an image processing model training method and a processing method, system, device and medium. Background Art

[0002] Image processing or image fusion technology is the core step of artificial intelligence recognition. By combining infrared images that capture thermal radiation information, information that is difficult for the naked eye to perceive can be obtained. Further, by combining the advantages of infrared images and visible light images, rich color information, detail information, and thermal radiation information can be obtained to fully analyze the current image.

[0003] However, the combination of infrared images and visible light images in existing technologies has not been widely recognized in terms of performance and combination satisfaction. Current image fusion technologies are mostly for images of the same type, and the technology for combining two different types of image information is not mature. Image fusion technology relies on the accuracy and richness of the fusion model, and the existing infrared image and visible light image models cannot meet people's daily needs.

[0004] Therefore, it is necessary to design a model suitable for fusing visible light images and infrared images. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes an image processing model training method and processing method, system, device and medium, aiming to solve the problem of lack of an effective method for fusing infrared images and visible light images in image processing models.

[0006] The technical solution proposed by the present invention is:

[0007] A method for training an image processing model, wherein the image processing model includes a plurality of hidden layers, and output information of a previous hidden layer is used as input information of a next hidden layer. The method for training an image processing model includes:

[0008] Using the visible light image and the infrared image of the same picture as input information of the initial hidden layer of the image processing model to obtain output information of the initial hidden layer;

[0009] Using the output information of the initial hidden layer as the input information of the next hidden layer to obtain the output information of the next hidden layer, until the target image output by the last hidden layer is obtained;

[0010] Calculating a first loss value according to the target image, the visible light image, and the infrared image;

[0011] Calculating a second loss value according to the target image and the visible light image;

[0012] Calculate the third loss value based on the information output by each hidden layer;

[0013] Update the parameters of the image processing model according to the first loss value, the second loss value and the third loss value.

[0014] Preferably, the image processing model includes a first loss function, and calculating the first loss value according to the target image, the visible light image and the infrared image includes:

[0015] Inputting the content information of the target image, the content information of the visible light image, and the content information of the infrared image into the first loss function to calculate a first loss value;

[0016] The first loss function formula is:

[0017]

[0018] in, is the first loss value, is the content information of the target image, The content information of the infrared image and is the content information of the visible light image.

[0019] Preferably, the image processing model includes a second loss function, and calculating the second loss value according to the target image and the visible light image includes:

[0020] Obtaining a first color histogram consisting of each pixel point and the corresponding pixel value in the target image;

[0021] Obtaining a second color histogram consisting of each pixel point and the corresponding pixel value in the visible light image;

[0022] Inputting the first color histogram and the second color histogram into the second loss function to calculate a second loss value;

[0023] The second loss function formula is:

[0024]

[0025] in, is the second loss value, is the horizontal coordinate of the image, is the vertical coordinate of the image, is the color histogram of the target image, is a color histogram of the visible light image.

[0026] Preferably, the image processing model includes a third loss function, and calculating the third loss value according to the output information of each hidden layer includes:

[0027] Input the output information of each hidden layer into the third loss function to calculate a third loss value;

[0028]

[0029] in, is the third loss value, For the The output information of the hidden layer, For the The output information of the hidden layer, ⊕ represents the idempotent operator. The idempotent operator uses addition operation.

[0030] Preferably, the image processing model includes a learning rate, and the image processing model training method includes:

[0031] Calculating a model loss rate according to current parameters and a learning rate of the image processing model;

[0032] Calculating and acquiring the parameters of the image processing model at the next moment according to the current parameters of the image processing model and the model loss rate;

[0033] Calculate a plurality of gradients according to the output of each hidden layer of the image processing model;

[0034] Parameters of the image processing model are updated according to the gradient.

[0035] The present application also provides an image processing method comprising:

[0036] Optimizing the size parameter and the resolution parameter of the infrared image or the visible light image according to the size parameter and the resolution parameter of the visible light image or the infrared image;

[0037] Using the optimized visible light image and infrared image as input information of a starting hidden layer of the image processing model as described in any one of the above items to obtain output information of the starting hidden layer, wherein the image processing model includes a plurality of hidden layers, and the output of the previous hidden layer is used as the input of the next hidden layer;

[0038] The output information of the initial hidden layer is used as the input information of the next hidden layer until a fused image output by the last hidden layer of the image processing model is obtained.

[0039] The present application also provides a training system for an image processing model, the training system comprising:

[0040] An acquisition module, used for taking the visible light image and the infrared image of the same picture as the input information of the initial hidden layer of the image processing model to acquire the output information of the initial hidden layer;

[0041] A fusion module is used to use the output information of the initial hidden layer as the input information of the next hidden layer to obtain the output information of the next hidden layer, until the target image output by the last hidden layer is obtained;

[0042] A first calculation module, used for calculating a first loss value according to the target image, the visible light image and the infrared image;

[0043] A second calculation module, used for calculating a second loss value according to the target image and the visible light image;

[0044] A third calculation module, used for calculating a third loss value according to the output information of each hidden layer;

[0045] A correction module is used to update the parameters of the image processing model according to the first loss value, the second loss value and the third loss value.

[0046] Preferably, the image processing model includes a first loss function, and the first calculation module includes:

[0047] A first loss module, used for inputting the content information of the target image, the content information of the visible light image and the content information of the infrared image into the first loss function to calculate a first loss value;

[0048] The first loss function formula is:

[0049]

[0050] in, is the first loss value, is the content information of the target image, The content information of the infrared image and is the content information of the visible light image.

[0051] Preferably, the image processing model includes a second loss function, and the second calculation module includes:

[0052] A first color picking module is used to obtain a first color histogram consisting of each pixel point and the corresponding pixel value in the target image;

[0053] A second color picking module is used to obtain a second color histogram consisting of each pixel point and the corresponding pixel value in the visible light image;

[0054] A second loss module, inputting the first color histogram and the second color histogram into the second loss function to calculate a second loss value;

[0055] The second loss function formula is:

[0056]

[0057] in, is the second loss value, is the horizontal coordinate of the image, is the vertical coordinate of the image, is the color histogram of the target image, is a color histogram of the visible light image.

[0058] Preferably, the image processing model includes a third loss function, and the third calculation module includes:

[0059] A third loss module, used for inputting the output information of each hidden layer into the third loss function to calculate a third loss value;

[0060]

[0061] in, is the third loss value, For the The output information of the hidden layer, For the The output information of the hidden layer, ⊕ represents the idempotent operator. The idempotent operator uses addition operation.

[0062] Preferably, the image processing model includes a learning rate, and the image processing model training method includes:

[0063] A learning module, calculating a model loss rate according to current parameters and a learning rate of the image processing model;

[0064] A first updating module is configured to calculate and obtain the parameters of the image processing model at the next moment according to the current parameters of the image processing model and the model loss rate;

[0065] A gradient module, which calculates a number of gradients according to the output of each hidden layer of the image processing model;

[0066] The second updating module updates the parameters of the image processing model according to the gradient.

[0067] The present application provides an image processing system, the image processing system comprising:

[0068] A preprocessing module, used to modify the size parameter and resolution parameter of the infrared image or the visible light image according to the size parameter and resolution parameter of the visible light image or the infrared image;

[0069] An input module, used to use the modified visible light image and infrared image as input information of a starting hidden layer of an image processing model to obtain output information of the starting hidden layer, wherein the image processing model includes a plurality of hidden layers, and the output of a previous hidden layer is used as the input of a next hidden layer;

[0070] The processing module is used to use the output information of the initial hidden layer as the input information of the next hidden layer until the fused image output by the last hidden layer of the image processing model is obtained.

[0071] In order to solve the above problems, an embodiment of the present invention also provides an electronic device, which includes: a memory and a processor, the memory stores a computer program, and the processor implements any one of the above-mentioned image processing model training methods and / or the above-mentioned image processing methods when executing the computer program.

[0072] In order to solve the above problems, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned image processing model training method and / or the above-mentioned image processing method.

[0073] According to the above technical solution, the beneficial effect of the present invention is as follows: the image processing model includes several hidden layers, the output information of the previous hidden layer is used as the input information of the next hidden layer, and the image processing model training method includes: using the visible light image and infrared image of the same picture as the input information of the starting hidden layer of the image processing model to obtain the output information of the starting hidden layer; using the output information of the starting hidden layer as the input information of the next hidden layer to obtain the output information of the next hidden layer, until the target image output by the last hidden layer is obtained; calculating the first loss value according to the target image, the visible light image and the infrared image; calculating the second loss value according to the target image and the visible light image; calculating the third loss value according to the information output by each hidden layer; updating the parameters of the image processing model according to the first loss value, the second loss value and the third loss value. The image processing model training effectively improves the fusion efficiency and accuracy of the visible light image and the infrared image by combining the image color, the image content and the idempotent operation model, and the idempotent operation model provides stability for the system, thereby improving the training efficiency. In addition, the present application also provides an image processing method, which can effectively fuse the visible light image and the infrared image together to generate a new analytical image with rich image content.

[0074] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A first flow chart of the image processing model training method provided in an embodiment of the present invention.

[0076] Figure 2 A schematic diagram of the first sub-process of the image processing model training method provided in an embodiment of the present invention.

[0077] Figure 3 A schematic diagram of the second sub-process of the image processing model training method provided in an embodiment of the present invention.

[0078] Figure 4 A schematic diagram of the third sub-process of the image processing model training method provided in an embodiment of the present invention.

[0079] Figure 5 A second flow chart of the image processing model training method provided in an embodiment of the present invention.

[0080] Figure 6 A schematic diagram of a flow chart of an image processing method provided by an embodiment of the present invention.

[0081] Figure 7 A schematic diagram of the structure of a training system for an image processing model provided in an embodiment of the present invention.

[0082] Figure 8 for Figure 7 A schematic structural diagram of the first computing module.

[0083] Fig. 9 for Figure 7 A schematic diagram of the structure of the second computing module.

[0084] Fig.10 for Figure 7 A schematic structural diagram of the third computing module.

[0085] Fig.11 for Figure 7 Schematic diagram of the substructure of the image processing model.

[0086] Fig.12 A schematic diagram of the structure of an image processing system provided by an embodiment of the present invention.

[0087] Fig.13 A schematic diagram of the internal structure of an electronic device provided for application of an embodiment of the present invention. DETAILED DESCRIPTION

[0088] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0089] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0090] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0091] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0092] In addition, in the description of the present specification and the appended claims, the terms "first" and "second" etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0093] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0094] An embodiment of the present invention provides an image processing model training method, and the execution subject of the image processing model training method includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the image processing model training method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The server can be an independent server, or it can be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0095] Please combine Figure 1 As shown, Figure 1 The first flow chart of the image processing model training method provided by the embodiment of the present invention. The image processing model includes several hidden layers, and the information output by the previous hidden layer is used as the input information of the next hidden layer. The image processing model training method includes:

[0096] Step S101, using the visible light image and the infrared image of the same picture as input information of the initial hidden layer of the image processing model to obtain output information of the initial hidden layer.

[0097] The image processing model is a model network including an idempotent generative network (IGN). In this embodiment, the input information is projected onto the target distribution or the target flow pattern, that is, the noise and real data in the input information are mapped to itself, thereby reducing the diffusion of the flow pattern and achieving multiple inputs without affecting the model itself. Specifically, the output calculation formula of the hidden layer is:

[0098]

[0099] in, It is The output of the hidden layer is It is The output of the hidden layer is and It is The weights and biases of the hidden layers, is an activation function, such as ReLU. ⊕ represents an idempotent operator. The idempotent operator uses a simple addition operation. and Through this design, the network can maintain stable output after multiple iterations.

[0100] Idempotent Generative Network (IGN) uses a special network structure, where the input layer receives noisy data and the output layer outputs the fused image. Specifically, the hidden layer of the IGN network uses idempotent operators for forward propagation to satisfy the property of "multiple applications do not change the output". This structure ensures that the network still maintains a stable output after multiple iterations. Idempotent operators usually use simple addition or multiplication operations.

[0101] In this embodiment, the idempotent generation network (IGN) can adopt solution 1, by comparing the noise or differences between the two graphs, superimposing these noise or differences on the third graph (intuitively); or solution 2 can be used to train the two graphs (including noise) into a pattern, and substitute the third graph into this pattern to obtain the fourth graph. Other feasible solutions can also be used.

[0102] Step S102, using the output information of the initial hidden layer as the input information of the next hidden layer to obtain the output information of the next hidden layer, until the target image output by the last hidden layer is obtained.

[0103] Among them, the visible light image and the infrared image are input into the starting hidden layer to obtain the information of the output of the starting hidden layer, and the output of each hidden layer is continuously used as the input of the next hidden layer until the output information of the last hidden layer is obtained as the target image.

[0104] Step S103, calculating a first loss value according to the target image, the visible light image and the infrared image.

[0105] Through the trained idempotent generative network, infrared images and visible light images can be fused to generate fused images with more realistic colors and richer information. This method of fusion images can better highlight the details in the image and improve the readability and recognition rate of the image.

[0106] More specifically, calculating the first loss value based on the target image, the visible light image, and the infrared image is described in detail below.

[0107] Step S104, calculating a second loss value according to the target image and the visible light image.

[0108] Color loss is used to measure the difference in color fidelity of the fused image, calculate the color value between the target image and the visible light image, and then balance the color difference between the target image and the real image, making the target image closer to the real color.

[0109] More specifically, calculating the second loss value based on the target image and the visible light image is described in detail below.

[0110] Step S105, calculating the third loss value according to the output information of each hidden layer.

[0111] The idempotent generation network uses idempotent operators to ensure that the network maintains stable output after multiple iterations. This design can reduce the network's sensitivity to input data and improve the stability of the fusion results.

[0112] More specifically, calculating the third loss value according to the output information of each hidden layer is described in detail below.

[0113] Step S106, updating the parameters of the image processing model according to the first loss value, the second loss value and the third loss value.

[0114]

[0115] is the loss function; is the first loss function, which represents the loss of image content; is the second loss function, which indicates the amount of loss of image color; is the third loss function, which represents the loss of idempotent operation; and is a hyperparameter used to balance the weights among various losses. and The dot product is between them. and The dot product is between.

[0116] During the training process, the sample data is substituted into the first loss function, the second loss function, and the third loss function to obtain the determined first loss value, second loss value, and third loss value, and then the sample data loss value of the image processing model is calculated through the loss function. The parameters of the image processing model are updated with the sample data loss value to minimize the loss value output by the image processing model.

[0117] In summary, the image processing model improves the quality of image fusion through the IGN network. The image processing model contains loss functions with content loss, color loss and idempotent loss. Content loss is used to measure the difference in content information between the fused image and the original infrared and visible light images; color loss is used to measure the difference in color fidelity of the fused image; idempotent loss is used to measure the stability of the network output. The three loss parts are balanced through hyperparameters to ensure that the network can simultaneously optimize content, color and stability during training. The image processing method that integrates idempotent operators improves the stability of the model during the iteration process. By fusing infrared images with visible light images, the target objects can be better identified and analyzed, and the efficiency and accuracy of related tasks can be improved.

[0118] See also Figure 2 As shown, Figure 2This is a schematic diagram of the first sub-process of the image processing model training method provided by an embodiment of the present invention. Step S102 includes:

[0119] Step S1021 , fusing the content information of the visible light image and the content information of the infrared image to obtain fused content information.

[0120] Step S1022: Subtract the content information of the target image from the fused content information to obtain difference information as the first loss value.

[0121] Inputting the content information of the target image, the content information of the visible light image, and the content information of the infrared image into a first loss function to calculate a first loss value;

[0122] The first loss function formula is:

[0123]

[0124] in, is the first loss value, is the content information of the target image, The content information of the infrared image and is the content information of the visible light image.

[0125] In this formula, The norm is used to measure the target image With combined images Specifically, it represents the square of the Euclidean distance between the two images. The norm calculates the sum of the squares of the differences between each pixel in the image, and then takes the square root, which reflects the overall difference in the image content. For example, a visible light image is the pixel position, chromaticity, color difference with adjacent pixels, and other content information of the pixel, and infrared content information is the temperature difference and radiation difference reflected by the color difference under the infrared spectrum.

[0126] See also Figure 3 As shown, Figure 3 This is a schematic diagram of a second sub-process of the image processing model training method provided by an embodiment of the present invention. Step S103 includes:

[0127] Step S1031, obtaining a first color histogram consisting of each pixel point and the corresponding pixel value in the target image.

[0128] Specifically, the first color histogram is a color histogram of the target image.

[0129] Step S1032: Obtain a second color histogram consisting of each pixel point and the corresponding pixel value in the visible light image.

[0130] Specifically, the second color histogram is a color histogram of the visible light image.

[0131] Step S1033: Obtain a second loss value according to the first color histogram and the second color histogram.

[0132] Specifically, the first color histogram and the second color histogram are input into the second loss function to calculate a second loss value.

[0133] The second loss function formula is:

[0134]

[0135] in, is the second loss value, is the horizontal coordinate of the image, is the vertical coordinate of the image, is the color histogram of the target image, is the color histogram of the visible light image. The color histogram is a color feature widely used in many image retrieval systems. It describes the proportion of different colors in the entire image, but does not care about the spatial position of each color, that is, it cannot describe the object or subject in the image.

[0136] The second loss function can explicitly optimize the color fidelity of the fused image, making the fused image closer to the real scene in color and improving the visual quality of the image.

[0137] See also Figure 4 As shown, Figure 4 This is a schematic diagram of a third sub-process of the image processing model training method provided by an embodiment of the present invention. Step S104 includes:

[0138] Step S1041, obtaining cumulative information by performing idempotent operations on information output by all hidden layers except the last hidden layer.

[0139] Step S1042, calculating a third loss value based on the accumulated information and the information output by the last hidden layer.

[0140] Specifically, the output information of each hidden layer is input into the third loss function to calculate the third loss value;

[0141] The third loss function formula:

[0142]

[0143] in, is the third loss value, For the The output information of the hidden layer, For the The output information of the hidden layer.

[0144] In this formula, The norm is used to measure the difference between the last hidden layer and the cumulative output of all previous hidden layers. Specifically, it represents the square of the Euclidean distance between the two images. The norm calculates the sum of the squares of the differences between each pixel between images and then takes the square root, reflecting the overall difference in the image content. Calculate the difference between the output of the last hidden layer and the idempotent result of the output of all previous hidden layers. By minimizing this loss, the image processing model can be encouraged to learn a more stable representation.

[0145] See also Figure 5 As shown, Figure 5 The second flow chart of the image processing model training method provided by the embodiment of the present invention is as follows. Step S105: The image processing model includes a learning rate, and the image processing model training method includes:

[0146] Step S1051, calculating the model loss rate according to the current parameters and learning rate of the image processing model.

[0147] Step S1052, calculating and obtaining the parameters of the image processing model at the next moment according to the current parameters of the image processing model and the model loss rate.

[0148] Step S1053, calculating a number of gradients according to the output of each hidden layer of the image processing model.

[0149] Step S1054, update the parameters of the image processing model according to the gradient.

[0150] During the training process, the system uses the stochastic gradient descent (SGD) algorithm to sample data and uses the back propagation algorithm to calculate the gradient of the loss function with respect to the network parameters. The parameter update rules are as follows:

[0151]

[0152] in, The network is at the moment Parameters, The network is at the moment Parameters, is the learning rate. It should be noted that the impact of the idempotent loss function on the hidden layer output needs to be considered when calculating the gradient. and The dot product is between them. Therefore, additional gradient terms need to be calculated to update the parameters of the hidden layer. Specifically, the gradient term calculation formula is:

[0153]

[0154] in, It is about The partial derivatives of the hidden layer outputs; It is about The output of the hidden layer. When calculating the gradient, it is necessary to consider the impact of the idempotent loss function on the hidden layer output, so additional gradient terms need to be calculated to update the parameters of the hidden layer. and The dot product is in between.

[0155] By calculating the gradient and updating the parameters of the hidden layer, the network can better learn stable representations. Techniques such as early stopping and model checkpointing are also used to prevent overfitting and improve training stability. During the training process, the stochastic gradient descent (SGD) algorithm is used to sample samples, and the back-propagation algorithm is used to calculate the gradient of the loss function with respect to the network parameters.

[0156] In addition, the proposed idempotent generation network structure is flexible and can be adjusted and improved according to specific application scenarios. The hyperparameters in the loss function can also be adjusted according to different usage scenarios to adapt to different fusion requirements, such as scenarios with high stability requirements, For scenes with strict color requirements, increase The weight of .

[0157] By combining the formula and loss function of the idempotent generative network and the optimization algorithm during training, high color fidelity infrared and visible light image fusion is achieved. This method can explicitly optimize color fidelity and improve the stability of network output during training, thereby obtaining a fused image with more realistic colors and richer information.

[0158] Please combine Figure 6 As shown, Figure 6 The present invention provides a flowchart of an image processing method according to an embodiment of the present invention. The image processing method comprises the following steps.

[0159] Step S201 : optimizing the size parameters and resolution parameters of the infrared image or the visible light image according to the size parameters and resolution parameters of the visible light image or the infrared image.

[0160] Specifically, the input visible light image and infrared image are converted into the same size parameters and resolution parameters, which can effectively improve the matching degree of the visible light image and the infrared image. This is achieved through interpolation algorithms such as bilinear interpolation and cubic spline interpolation or histogram equalization. The input visible light image and infrared image are converted into standardized images and used as subsequent inputs.

[0161] Step S202: Using the optimized visible light image and infrared image as input information of the initial hidden layer of the above-mentioned image processing model to obtain output information of the initial hidden layer.

[0162] The image processing model includes several hidden layers, and the output of the previous hidden layer serves as the input of the next hidden layer.

[0163] Step S203, using the output information of the initial hidden layer as the input information of the next hidden layer, until the fused image output by the last hidden layer of the image processing model is obtained.

[0164] In order to improve the visual quality of the generated fused image, the system also adds post-processing steps including color correction and contrast enhancement. Specifically, the color correction algorithm can be used to adjust the color balance and saturation of the fused image to make it closer to the color representation of the real scene. In addition, the contrast enhancement algorithm can be used to enhance the contrast of the fused image and highlight the details in the image. These post-processing steps can select different algorithms and implementation methods according to actual needs.

[0165] Please combine Figure 7 As shown, Figure 7 The structural diagram of a training system for an image processing model provided by an embodiment of the present invention is as follows: The training system 10 for an image processing model comprises a first acquisition module 101 , a fusion module 102 , a first calculation module 103 , a second calculation module 104 , a third calculation module 105 and a correction module 106 .

[0166] The acquisition module 101 is used to use the visible light image and the infrared image of the same picture as the input information of the initial hidden layer of the image processing model to obtain the output information of the initial hidden layer.

[0167] The image processing model is a model network including an idempotent generative network (IGN). In this embodiment, the input information is projected onto the target distribution or the target flow pattern, that is, the noise and real data in the input information are mapped to itself, thereby reducing the diffusion of the flow pattern and achieving multiple inputs without affecting the model itself. Specifically, the output calculation formula of the hidden layer is:

[0168]

[0169] in, It is The output of the hidden layer is It is The output of the hidden layer is and It is The weights and biases of the hidden layers, is an activation function, such as ReLU. ⊕ represents an idempotent operator. The idempotent operator uses a simple addition operation. and Through this design, the network can maintain stable output after multiple iterations.

[0170] Idempotent Generative Network (IGN) uses a special network structure, where the input layer receives noisy data and the output layer outputs the fused image. Specifically, the hidden layer of the IGN network uses idempotent operators for forward propagation to satisfy the property of "multiple applications do not change the output". This structure ensures that the network still maintains a stable output after multiple iterations. Idempotent operators usually use simple addition or multiplication operations.

[0171] In this embodiment, the idempotent generation network (IGN) can adopt solution 1, by comparing the noise or differences between the two graphs, superimposing these noise or differences on the third graph (intuitively); or solution 2 can be used to train the two graphs (including noise) into a pattern, and substitute the third graph into this pattern to obtain the fourth graph. Other feasible solutions can also be used.

[0172] The fusion module 102 is used to use the output information of the initial hidden layer as the input information of the next hidden layer to obtain the output information of the next hidden layer, until the target image output by the last hidden layer is obtained.

[0173] Among them, the visible light image and the infrared image are input into the starting hidden layer to obtain the information of the output of the starting hidden layer, and the output of each hidden layer is continuously used as the input of the next hidden layer until the output information of the last hidden layer is obtained as the target image.

[0174]

[0175] is the loss function; is the first loss function, which represents the loss of image content; is the second loss function, which indicates the amount of loss of image color; is the third loss function, which represents the loss of idempotent operation; and is a hyperparameter used to balance the weights among various losses. and The dot product is between them. and The dot product is in between.

[0176] The first calculation module 103 is used to calculate a first loss value according to the target image, the visible light image and the infrared image.

[0177] Through the trained idempotent generative network, infrared images and visible light images can be fused to generate fused images with more realistic colors and richer information. This method of fusion images can better highlight the details in the image and improve the readability and recognition rate of the image.

[0178] The second calculation module 104 is used to calculate a second loss value according to the target image and the visible light image.

[0179] Color loss is used to measure the difference in color fidelity of the fused image, calculate the color value between the target image and the visible light image, and then balance the color difference between the target image and the real image, making the target image closer to the real color.

[0180] The third calculation module 105 is used to calculate the third loss value according to the output information of each hidden layer.

[0181] The idempotent generation network uses idempotent operators to ensure that the network maintains stable output after multiple iterations. This design can reduce the network's sensitivity to input data and improve the stability of the fusion results.

[0182] The correction module 106 is used to update the parameters of the image processing model according to the first loss value, the second loss value and the third loss value.

[0183] In summary, the image processing model improves the quality of image fusion through the IGN network. The image processing model contains loss functions with content loss, color loss and idempotent loss. Content loss is used to measure the difference in content information between the fused image and the original infrared and visible light images; color loss is used to measure the difference in color fidelity of the fused image; idempotent loss is used to measure the stability of the network output. The three loss parts are balanced through hyperparameters to ensure that the network can simultaneously optimize content, color and stability during training. The image processing method that integrates idempotent operators improves the stability of the model during the iteration process. By fusing infrared images with visible light images, the target objects can be better identified and analyzed, and the efficiency and accuracy of related tasks can be improved.

[0184] Please combine Figure 8 As shown, Figure 8 A schematic diagram of the structure of the first calculation module provided in an embodiment of the present invention. The image processing model includes a first loss function, and the first calculation module 103 includes:

[0185] The first loss module 1301 is used to input the content information of the target image, the content information of the visible light image and the content information of the infrared image into a first loss function to calculate a first loss value.

[0186] Specifically, the fused content information is obtained by fusing the content information of the visible light image and the content information of the infrared image, and the difference information is obtained by subtracting the content information of the target image from the fused content information as the first loss value.

[0187] The first loss function formula is:

[0188]

[0189] in, is the first loss value, is the content information of the target image, The content information of the infrared image and is the content information of the visible light image.

[0190] For example, a visible light image includes the pixel position, chromaticity, color difference with adjacent pixels, and other content information of the pixel, while infrared content information includes the temperature difference and radiation difference of the color difference reaction under the infrared spectrum.

[0191] Please combine Fig. 9 As shown, Fig. 9 The schematic diagram of the structure of the second calculation module provided by the embodiment of the present invention. The image processing model includes a second loss function, and the second calculation module 104 includes:

[0192] The first color picking module 1401 obtains a first color histogram consisting of each pixel point and the corresponding pixel value in the target image.

[0193] The second color picking module 1402 obtains a second color histogram consisting of each pixel point and the corresponding pixel value in the visible light image.

[0194] The second loss module 1403 inputs the first color histogram and the second color histogram into a second loss function to calculate a second loss value.

[0195] The second loss function formula is:

[0196]

[0197] in, is the second loss value, is the horizontal coordinate of the image, is the vertical coordinate of the image, is the color histogram of the target image, is the color histogram of the visible light image. The color histogram is a color feature widely used in many image retrieval systems. It describes the proportion of different colors in the entire image, but does not care about the spatial position of each color, that is, it cannot describe the object or subject in the image.

[0198] Please combine Fig.10 As shown, Fig.10 A schematic diagram of the structure of the third calculation module provided in an embodiment of the present invention. The image processing model includes a third loss function, and the third calculation module 105 includes:

[0199] The third loss module 1051 is used to input the output information of each hidden layer into the third loss function to calculate the third loss value.

[0200] Specifically, the information output by all hidden layers except the last hidden layer is used to obtain cumulative information through idempotent operation, and the third loss value is calculated according to the cumulative information and the information output by the last hidden layer.

[0201]

[0202] in, is the third loss value, For the The output information of the hidden layer, For the The output information of the hidden layer.

[0203] Please combine Fig.11 As shown, Fig.11 A schematic diagram of a substructure of an image processing model provided by an embodiment of the present invention. The image processing model includes a learning rate, and the training system 10 of the image processing model also includes:

[0204] The learning module 107 calculates the model loss rate according to the current parameters of the image processing model and the learning rate.

[0205] The first updating module 108 calculates and obtains the parameters of the image processing model at the next moment according to the current parameters of the image processing model and the model loss rate.

[0206] During the training process, the system uses the stochastic gradient descent (SGD) algorithm to sample samples and uses the back propagation algorithm to calculate the gradient of the loss function with respect to the network parameters. The parameter update rules are as follows:

[0207]

[0208] in, The network is at the moment Parameters, The network is at the moment Parameters, is the learning rate. It should be noted that the impact of the idempotent loss function on the hidden layer output needs to be considered when calculating the gradient. and Therefore, additional gradient terms need to be calculated to update the parameters of the hidden layer.

[0209] The gradient module 109 calculates a number of gradients according to the output of each hidden layer of the image processing model.

[0210] The second updating module 110 updates the parameters of the image processing model according to the gradient.

[0211] Specifically, the gradient term is calculated as:

[0212]

[0213] in, It is about The partial derivatives of the hidden layer outputs; It is about The output of the hidden layer. When calculating the gradient, it is necessary to consider the impact of the idempotent loss function on the hidden layer output, so additional gradient terms need to be calculated to update the parameters of the hidden layer. and The dot product is in between.

[0214] By calculating the gradient and updating the parameters of the hidden layer, the network can better learn stable representations. Techniques such as early stopping and model checkpointing are also used to prevent overfitting and improve training stability. During the training process, the stochastic gradient descent (SGD) algorithm is used to sample samples, and the back-propagation algorithm is used to calculate the gradient of the loss function with respect to the network parameters.

[0215] In addition, the proposed idempotent generation network structure is flexible and can be adjusted and improved according to specific application scenarios. The hyperparameters in the loss function can also be adjusted according to different usage scenarios to adapt to different fusion requirements, such as scenarios with high stability requirements, For scenes with strict color requirements, increase The weight of .

[0216] By combining the formula and loss function of the idempotent generative network and the optimization algorithm during training, high color fidelity infrared and visible light image fusion is achieved. This method can explicitly optimize color fidelity and improve the stability of network output during training, thereby obtaining a fused image with more realistic colors and richer information.

[0217] Please combine Fig.12 As shown, Fig.12 The image processing system 20 is a schematic diagram of the structure of the image processing system provided by the embodiment of the present invention. The image processing system 20 includes:

[0218] The preprocessing module 201 is used to modify the size parameters and resolution parameters of the infrared image or the visible light image according to the size parameters and resolution parameters of the visible light image or the infrared image.

[0219] Specifically, the input visible light image and infrared image are converted into the same size parameters and resolution parameters, which can effectively improve the matching degree of the visible light image and the infrared image. This is achieved through interpolation algorithms such as bilinear interpolation and cubic spline interpolation or histogram equalization. The input visible light image and infrared image are converted into standardized images and used as subsequent inputs.

[0220] The input module 202 is used to use the modified visible light image and infrared image as input information of the initial hidden layer of the image processing model to obtain output information of the initial hidden layer.

[0221] The image processing model includes several hidden layers, and the output of the previous hidden layer serves as the input of the next hidden layer.

[0222] The processing module 203 is used to use the output information of the initial hidden layer as the input information of the next hidden layer until the fused image output by the last hidden layer of the image processing model is obtained.

[0223] In order to improve the visual quality of the generated fused image, the system adds post-processing steps including color correction and contrast enhancement. Specifically, the color correction algorithm can be used to adjust the color balance and saturation of the fused image to make it closer to the color representation of the real scene. In addition, the contrast enhancement algorithm can be used to enhance the contrast of the fused image and highlight the details in the image. These post-processing steps can select different algorithms and implementation methods according to actual needs.

[0224] In detail, each module in the training system 10 of the image processing model in the embodiment of the present invention adopts the same method as above when used. Figure 1-Figure 5 The image processing model training method in the above method is the same as the technical means and can produce the same technical effects, so I will not go into details here.

[0225] In detail, each module in the image processing system 20 in the embodiment of the present invention is used in the same manner as described above. Figure 1-Figure 5 The model trained by the image processing model training method in Figure 6 The image processing method has the same technical means and can produce the same technical effects, so it will not be repeated here.

[0226] The present invention also discloses an electronic device 1000, see Fig.13 As shown, Fig.13 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention.

[0227] The electronic device 1000 may include at least one processor 1; and a memory 2 in communication with the at least one processor 1. The memory 2 stores a computer program executable by the at least one processor 1, and the computer program is executed by the at least one processor 1 so that the at least one processor 1 can perform the above-mentioned image processing model training method and / or the above-mentioned image processing method.

[0228] In some embodiments, the processor 1 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 1 is the control core (ControlUnit) of the electronic device 1000, and uses various interfaces and lines to connect various components of the entire electronic device 1000, and executes or executes programs or modules (such as image processing programs, etc.) stored in the memory 2, and calls data stored in the memory 2 to execute various functions of the electronic device 1000 and process data.

[0229] Furthermore, if the module / unit integrated in the electronic device 1000 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable storage medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0230] The present invention also provides a computer-readable storage medium, including a computer program, which can be executed by a processor 1 to complete an image processing model training method and / or an image processing method.

[0231] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0232] Therefore, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0233] It should be understood that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for training an image processing model, characterized in that: The image processing model includes several hidden layers, and the output information of the previous hidden layer is used as the input information of the next hidden layer. The image processing model training method includes: Using the visible light image and the infrared image of the same picture as input information of the initial hidden layer of the image processing model to obtain output information of the initial hidden layer; Using the output information of the initial hidden layer as the input information of the next hidden layer to obtain the output information of the next hidden layer, until the target image output by the last hidden layer is obtained; Calculating a first loss value according to the target image, the visible light image, and the infrared image; The image processing model includes a first loss function, and calculating the first loss value according to the target image, the visible light image, and the infrared image includes: Inputting the content information of the target image, the content information of the visible light image, and the content information of the infrared image into the first loss function to calculate a first loss value; The first loss function formula is: in, is the first loss value, is the content information of the target image, The content information of the infrared image and is the content information of the visible light image; Calculating a second loss value according to the target image and the visible light image; The image processing model includes a second loss function, and calculating the second loss value according to the target image and the visible light image includes: Obtaining a first color histogram consisting of each pixel point and the corresponding pixel value in the target image; Obtaining a second color histogram consisting of each pixel point and the corresponding pixel value in the visible light image; Inputting the first color histogram and the second color histogram into the second loss function to calculate a second loss value; The second loss function formula is: in, is the second loss value, is the horizontal coordinate of the image, is the vertical coordinate of the image, is the color histogram of the target image, is a color histogram of the visible light image; Calculate the third loss value based on the information output by each hidden layer; The image processing model includes a third loss function, and calculating the third loss value according to the information output by each hidden layer includes: Input the information output by each hidden layer into the third loss function to calculate a third loss value; in, is the third loss value, For the The output information of the hidden layer, For the The output information of the hidden layer, ⊕ represents the idempotent operator, and the idempotent operator uses addition operation; Update the parameters of the image processing model according to the first loss value, the second loss value and the third loss value.

2. The image processing model training method according to claim 1, characterized in that: The image processing model includes a learning rate, and the image processing model training method includes: Calculating a model loss rate according to current parameters and a learning rate of the image processing model; Calculating and acquiring the parameters of the image processing model at the next moment according to the current parameters of the image processing model and the model loss rate; Calculate a plurality of gradients according to the output of each hidden layer of the image processing model; Parameters of the image processing model are updated according to the gradient.

3. An image processing method, characterized in that: The image processing method comprises: Optimizing the size parameter and the resolution parameter of the infrared image or the visible light image according to the size parameter and the resolution parameter of the visible light image or the infrared image; Using the optimized visible light image and infrared image as input information of a starting hidden layer of the image processing model according to any one of claims 1 to 2 to obtain output information of the starting hidden layer, wherein the image processing model includes a plurality of hidden layers, and the output of a previous hidden layer is used as the input of a next hidden layer; The output information of the initial hidden layer is used as the input information of the next hidden layer until a fused image output by the last hidden layer of the image processing model is obtained.

4. A training system for an image processing model, characterized in that: The training system comprises: An acquisition module, used for taking the visible light image and the infrared image of the same picture as the input information of the initial hidden layer of the image processing model to acquire the output information of the initial hidden layer; A fusion module is used to use the output information of the initial hidden layer as the input information of the next hidden layer to obtain the output information of the next hidden layer, until the target image output by the last hidden layer is obtained; A first calculation module, used for calculating a first loss value according to the target image, the visible light image and the infrared image; The image processing model includes a first loss function, and the first calculation module includes: A first loss module, used for inputting the content information of the target image, the content information of the visible light image and the content information of the infrared image into the first loss function to calculate a first loss value; The first loss function formula is: in, is the first loss value, is the content information of the target image, The content information of the infrared image and is the content information of the visible light image; A second calculation module, used for calculating a second loss value according to the target image and the visible light image; The image processing model includes a second loss function, and the second calculation module includes: A first color picking module is used to obtain a first color histogram consisting of each pixel point and the corresponding pixel value in the target image; A second color picking module is used to obtain a second color histogram consisting of each pixel point and the corresponding pixel value in the visible light image; A second loss module, inputting the first color histogram and the second color histogram into the second loss function to calculate a second loss value; The second loss function formula is: in, is the second loss value, is the horizontal coordinate of the image, is the vertical coordinate of the image, is the color histogram of the target image, is a color histogram of the visible light image; A third calculation module, used for calculating a third loss value according to the output information of each hidden layer; The image processing model includes a third loss function, and the third calculation module includes: A third loss module, used for inputting the output information of each hidden layer into the third loss function to calculate a third loss value; in, is the third loss value, For the The output information of the hidden layer, For the The output information of the hidden layer, ⊕ represents the idempotent operator, and the idempotent operator uses addition operation; A correction module is used to update the parameters of the image processing model according to the first loss value, the second loss value and the third loss value.

5. An image processing system, characterized in that: The image processing system adopts the image processing model trained by the training system as claimed in claim 4, comprising: A preprocessing module, used to modify the size parameter and resolution parameter of the infrared image or the visible light image according to the size parameter and resolution parameter of the visible light image or the infrared image; An input module, used to use the modified visible light image and infrared image as input information of a starting hidden layer of an image processing model to obtain output information of the starting hidden layer, wherein the image processing model includes a plurality of hidden layers, and the output of a previous hidden layer is used as the input of a next hidden layer; The processing module is used to use the output information of the initial hidden layer as the input information of the next hidden layer until the fused image output by the last hidden layer of the image processing model is obtained.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Citation Information

Patent Citations

  • Infrared light and visible light image fusion method based on residual dense network and gradient loss

    CN113592018A

  • Target detection method and system based on infrared and visible light image fusion

    CN117392496A

  • Data conversion method and device, model training method, equipment and medium

    CN117710196A

  • Infrared and visible light image fusion detection method based on interactive optimization strategy

    CN117893422A

  • Image fusion method and apparatus and training method and apparatus for image fusion model

    US20230214976A1