Image processing method and system

Abstract

The invention provides an image processing method and system, and belongs to the technical field of image fusion. According to the method, from the perspective of a network channel, the characteristics of Hankel convolution expanded by a Hankel matrix are combined, the method is inspired by GSVD, and a novel data interaction model is provided. Different from traditional shared parameters between channels, data of different channels can be flexibly switched by using Hankel convolution, so that data interaction is realized. Through interaction of the data, the feature extraction level of multiple inputs is improved, so that the defects of a multi-path network parallel feature extraction mode are overcome, and more efficient processing efficiency is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion, and in particular relates to an image processing method and system. Background Art

[0002] The application of deep learning in multi-input tasks is very extensive and of great significance. Multi-input tasks involve processing multiple different or similar input data, and completing various tasks through subsequent processing. However, for the processing of multiple inputs, most existing methods use one channel for processing after the input, and then uniformly input the output of each network into the subsequent model to achieve the expected effect. This method has achieved good results in many tasks, such as CT image reconstruction, multimodal fusion, etc. Compared with traditional methods, deep learning can achieve higher precision processing, and has obvious advantages in big data situations.

[0003] However, when this network form is used for processing with multiple networks separately, there is no close connection between channels, which will lead to the lack of comprehensive consideration of multiple input data in the feature extraction process, reducing data utilization efficiency. Therefore, from the perspective of channel interaction, it is very meaningful to study a robust channel interaction module to improve data utilization.

[0004] From the existing network model, taking Transformer as an example, by using Transformer to establish a channel attention mechanism, reasonable weights are provided for the features of each channel to achieve the connection between channels. Its main components are:

[0005] Self-Attention: This is one of the core concepts of Transformer, which enables the model to consider all positions in the input sequence at the same time, rather than processing them step by step like Recurrent Neural Networks (RNN) or Convolutional Neural Networks (CNN). The self-attention mechanism allows the model to give different attention weights to different parts of the input sequence, thereby better capturing semantic relationships.

[0006] Multi-Head Attention: The self-attention mechanism in Transformer is extended to multiple attention heads, each of which can learn different attention weights to better capture different types of relationships. Multi-head attention allows the model to process different information subspaces in parallel.

[0007] Encoder and Decoder: Transformer usually consists of an encoder for processing the input sequence and a decoder for generating the output sequence, which makes it suitable for sequence-to-sequence tasks such as machine translation.

[0008] The above processing methods cannot effectively establish connections between data during feature extraction and fully utilize the advantages of multiple input data. In addition, the combined application of multiple networks will also increase the difficulty of network model training. Summary of the invention

[0009] The present invention aims at solving the deficiencies in the prior art and proposes an image processing solution.

[0010] A first aspect of the present invention provides an image processing method, the method comprising:

[0011] Step S1, acquiring an infrared image and a visible light image;

[0012] Among them, M infrared images and corresponding M visible light images are selected as infrared image training set and visible light image training set respectively; N infrared images and corresponding N visible light images are selected as infrared image test set and visible light image test set respectively; M>N;

[0013] Step S2, constructing an image processing system;

[0014] The image processing system includes a data interaction module and an image fusion network. The data interaction module includes an interactive element calculation module and four Hankel convolution modules. The image fusion network is an IFCNN network. The data interaction module and the image fusion network are connected in series.

[0015] Step S3, training the image processing system using the infrared image training set and the visible light image training set, and testing the trained image processing system using the infrared image test set and the visible light image test set;

[0016] Among them, the loss function in the training process is calculated using pixel loss and gradient loss.

[0017] According to the method of the first aspect, the data interaction module includes two channels, and two Hankel convolution modules are respectively configured in the first channel and the second channel; wherein, during the training / testing process:

[0018] The data interaction module receives an input infrared grayscale image and a visible light grayscale image;

[0019] In the first channel, the infrared grayscale image passes through the first Hankel convolution module, the interactive element calculation module and the second Hankel convolution module respectively to obtain infrared image data after interactive processing;

[0020] In the second channel, the visible light grayscale image passes through the third Hankel convolution module, the interactive element calculation module and the third Hankel convolution module respectively to obtain the visible light image data after interactive processing.

[0021] According to the method of the first aspect, P1 and P2 represent an infrared grayscale image and a visible light grayscale image respectively; the output of the first Hankel convolution module is The output of the third Hankel convolution module is Represents the initialized convolutional basis; the interactive element calculation module uses the interactive element B 1,2 To interactively switch the output of the first Hankel convolution module and the output of the third Hankel convolution module, the interactive switching process is described as: The outputs of the interactive element calculation module in the first channel and the second channel are sent to the second Hankel convolution module and the fourth Hankel convolution module respectively.

[0022] According to the method of the first aspect, the loss function in the training process is calculated using pixel loss and gradient loss, which is specifically expressed as:

[0023]

[0024] Among them, I i represents the infrared images in the infrared image training set, I v represents the visible light image in the visible light image training set, and the fused image output by the image fusion network is I f .

[0025] The second aspect of the present invention provides an image processing system, the system comprising: a data interaction module and an image fusion network, the data interaction module comprising an interactive element calculation module and four Hankel convolution modules, the image fusion network is an IFCNN network, and the data interaction module and the image fusion network are connected in series; wherein:

[0026] Acquire infrared images and visible light images;

[0027] Select M infrared images and corresponding M visible light images as infrared image training set and visible light image training set respectively; select N infrared images and corresponding N visible light images as infrared image test set and visible light image test set respectively; M>N;

[0028] The image processing system is trained using the infrared image training set and the visible light image training set, and the trained image processing system is tested using the infrared image test set and the visible light image test set;

[0029] The loss function during training is calculated using pixel loss and gradient loss.

[0030] According to the system of the second aspect, the data interaction module includes two channels, and two Hankel convolution modules are respectively configured in the first channel and the second channel; wherein, during the training / testing process:

[0031] The data interaction module receives an input infrared grayscale image and a visible light grayscale image;

[0032] In the first channel, the infrared grayscale image passes through the first Hankel convolution module, the interactive element calculation module and the second Hankel convolution module respectively to obtain infrared image data after interactive processing;

[0033] In the second channel, the visible light grayscale image passes through the third Hankel convolution module, the interactive element calculation module and the third Hankel convolution module respectively to obtain the visible light image data after interactive processing.

[0034] According to the system of the second aspect, P1 and P2 represent an infrared grayscale image and a visible light grayscale image respectively; the output of the first Hankel convolution module is The output of the third Hankel convolution module is Represents the initialized convolutional basis; the interactive element calculation module uses the interactive element B 1,2 To interactively switch the output of the first Hankel convolution module and the output of the third Hankel convolution module, the interactive switching process is described as: The outputs of the interactive element calculation module in the first channel and the second channel are sent to the second Hankel convolution module and the fourth Hankel convolution module respectively.

[0035] According to the system of the second aspect, the loss function in the training process is calculated using pixel loss and gradient loss, which is specifically expressed as:

[0036]

[0037] Among them, I i represents the infrared images in the infrared image training set, I v represents the visible light image in the visible light image training set, and the fused image output by the image fusion network is I f .

[0038] The third aspect of the present invention discloses an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the image processing method described in the first aspect of the present disclosure are implemented.

[0039] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the image processing method described in the first aspect of the present disclosure are implemented.

[0040] In summary, the present invention proposes a relatively novel data interaction model from the perspective of network channels, combined with the characteristics of Hankel convolution extended by Hankel matrix, inspired by GSVD. Different from the traditional shared parameters between channels, by using Hankel convolution, data from different channels can be flexibly switched to achieve data interaction. Through this interaction between data, the feature extraction level of multiple inputs is improved to overcome the shortcomings of multi-channel network parallel feature extraction methods, thereby achieving more efficient processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 Schematic diagram of the structure of a data processing system according to an embodiment of the present invention.

[0043] Figure 2 Schematic diagram of a data interaction module according to an embodiment of the present invention.

[0044] Figure 3 Schematic diagram of an interactive switching process according to an embodiment of the present invention.

[0045] Figure 4 FIG. 4 is a schematic diagram of image fusion according to an embodiment of the present invention.

[0046] Figure 5 The figure is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] The first aspect of the present invention provides an image processing method, the method comprising (such as Figure 1 shown):

[0049] Step S1, acquiring an infrared image and a visible light image;

[0050] Among them, M infrared images and corresponding M visible light images are selected as infrared image training set and visible light image training set respectively; N infrared images and corresponding N visible light images are selected as infrared image test set and visible light image test set respectively; M>N;

[0051] Step S2, constructing an image processing system;

[0052] The image processing system includes a data interaction module and an image fusion network. The data interaction module includes an interactive element calculation module and four Hankel convolution modules. The image fusion network is an IFCNN network. The data interaction module and the image fusion network are connected in series.

[0053] Step S3, training the image processing system using the infrared image training set and the visible light image training set, and testing the trained image processing system using the infrared image test set and the visible light image test set;

[0054] Among them, the loss function in the training process is calculated using pixel loss and gradient loss.

[0055] According to the method of the first aspect, the data interaction module includes two channels, and two Hankel convolution modules are respectively configured in the first channel and the second channel; wherein, during the training / testing process:

[0056] The data interaction module receives an input infrared grayscale image and a visible light grayscale image;

[0057] In the first channel, the infrared grayscale image passes through the first Hankel convolution module, the interactive element calculation module and the second Hankel convolution module respectively to obtain infrared image data after interactive processing;

[0058] In the second channel, the visible light grayscale image passes through the third Hankel convolution module, the interactive element calculation module and the third Hankel convolution module respectively to obtain the visible light image data after interactive processing.

[0059] According to the method of the first aspect, P1 and P2 represent an infrared grayscale image and a visible light grayscale image respectively; the output of the first Hankel convolution module is The output of the third Hankel convolution module is Represents the initialized convolutional basis; the interactive element calculation module uses the interactive element B1,2 To interactively switch the output of the first Hankel convolution module and the output of the third Hankel convolution module, the interactive switching process is described as: The outputs of the interactive element calculation module in the first channel and the second channel are sent to the second Hankel convolution module and the fourth Hankel convolution module respectively.

[0060] According to the method of the first aspect, the loss function in the training process is calculated using pixel loss and gradient loss, which is specifically expressed as:

[0061]

[0062] Among them, I i represents the infrared images in the infrared image training set, I v represents the visible light image in the visible light image training set, and the fused image output by the image fusion network is I f .

[0063] In some embodiments, the proposed network interaction module can realize data interaction and transmission between two channels. The data interaction module mainly consists of two parts:

[0064] (1) Interactive meta-computing process (including interaction and switching), such as Figure 2-3 As shown in the figure, the Combine part constructs an interactive element through two input data to achieve switching and interaction between network input data; (2) The network propagation process mainly includes two convolutions. The first convolution is a conventional Hankel convolution, and the second convolution is performed with the interactive element to achieve data interaction.

[0065] In addition, the data interaction module and other models can generally adopt a series connection mechanism, and subsequent processing can still be divided into two networks for transmission.

[0066] In some embodiments, the interaction module provides a solution for establishing connections between networks. Taking two inputs as an example, the basic definition of Hankel convolution is given first:

[0067] Input P∈R nl×p And the convolution basis Ψ∈R p×q , each column of the input is considered as a vector of n dimensions:

[0068]

[0069] Hankel convolution can be expressed as:

[0070]

[0071] in, Indicates that the matrix is ​​flipped column by column. for:

[0072]

[0073] Calculate the interactive element and construct the interactive element through two inputs P1 and P2. First, transpose the input divided by the column vector. Different from the general transposition method, each small vector is transposed separately in the calculation, that is:

[0074]

[0075] Then the transposed result and Inverse, we get and satisfy When the matrix does not have an inverse, it can be replaced by a generalized inverse, or the image can be segmented and then input into the interaction module separately, and then the output can be reassembled. The interaction element is calculated as:

[0076]

[0077] Initialize the convolutional basis Ψ and operate on the two inputs respectively, expressed as:

[0078]

[0079] Through the interactive element, data can be switched and interacted. The interactive element can switch the input data that plays a role in a certain network. 1,2 B 1,2 When , the switching can be expressed as:

[0080]

[0081] After the interaction is completed, the output is obtained, and the switched data is subsequently processed according to the task requirements.

[0082] The interactive module proposed in the present invention can effectively build a bridge between data, thereby realizing the interaction between network channels to improve the efficiency of multi-input networks. The module has high flexibility and can be added at the front end, middle and back end of the network, depending on the specific situation.

[0083] In some embodiments, in order to verify the effectiveness of the data interaction method provided by the proposed interaction module, a multimodal image (infrared, visible light) fusion experiment is used for verification. The more classic image fusion network IFCNN is used as the basic network, and the interaction module and the IFCNN model are connected in series. In order to more intuitively reflect the gain of infrared and visible light fusion, the preprocessing method of multiple channels is not used before input like the original model, but both images are converted into grayscale images and input into the network. The specific parameters are shown in Table 1.

[0084] Table 1: Main technical parameters used

[0085]

[0086] After selecting FILR and registering, 1000 images are selected as the training set, and then the TNO dataset is selected as the test set. Pixel loss and gradient loss are selected as the loss function, and the visible light image is I v , the infrared image is I i , the output fused image is I f , the loss function can be expressed as:

[0087]

[0088] The fusion result is Figure 4 As shown in the figure, it can be found that the network with the interactive module has obvious advantages in detail recovery. Compared with the use of IFCNN alone, the interactive module can retain clearer textures while ensuring the fusion effect, which can improve the subsequent detection and recognition accuracy. However, after a large number of experiments, it was found that when there are huge differences in the input, the addition of the interactive module can still have advantages in details, but it may have a certain counter-effect on some image blocks, so some images need to be preprocessed in advance to reduce the differences between images and improve network performance.

[0089] In summary, the present invention provides a new interaction mode for multi-input networks, builds a bridge between channels, and comprehensively utilizes the characteristics of multiple sets of data: combining the proposed interaction module with the network can effectively improve the network effect, establish connections between channels, and thus find common features in high-dimensional space; the interaction structure proposed by this technology is more flexible in application, and can fully interact at different positions in the network, thereby responding to more diverse task requirements.

[0090] The second aspect of the present invention provides an image processing system, the system comprising: a data interaction module and an image fusion network, the data interaction module comprising an interactive element calculation module and four Hankel convolution modules, the image fusion network is an IFCNN network, and the data interaction module and the image fusion network are connected in series; wherein:

[0091] Acquire infrared images and visible light images;

[0092] Select M infrared images and corresponding M visible light images as infrared image training set and visible light image training set respectively; select N infrared images and corresponding N visible light images as infrared image test set and visible light image test set respectively; M>N;

[0093] The image processing system is trained using the infrared image training set and the visible light image training set, and the trained image processing system is tested using the infrared image test set and the visible light image test set;

[0094] The loss function during training is calculated using pixel loss and gradient loss.

[0095] According to the system of the second aspect, the data interaction module includes two channels, and two Hankel convolution modules are respectively configured in the first channel and the second channel; wherein, during the training / testing process:

[0096] The data interaction module receives an input infrared grayscale image and a visible light grayscale image;

[0097] In the first channel, the infrared grayscale image passes through the first Hankel convolution module, the interactive element calculation module and the second Hankel convolution module respectively to obtain infrared image data after interactive processing;

[0098] In the second channel, the visible light grayscale image passes through the third Hankel convolution module, the interactive element calculation module and the third Hankel convolution module respectively to obtain the visible light image data after interactive processing.

[0099] According to the system of the second aspect, P1 and P2 represent an infrared grayscale image and a visible light grayscale image respectively; the output of the first Hankel convolution module is The output of the third Hankel convolution module is Represents the initialized convolutional basis; the interactive element calculation module uses the interactive element B 1,2 To interactively switch the output of the first Hankel convolution module and the output of the third Hankel convolution module, the interactive switching process is described as: The outputs of the interactive element calculation module in the first channel and the second channel are sent to the second Hankel convolution module and the fourth Hankel convolution module respectively.

[0100] According to the system of the second aspect, the loss function in the training process is calculated using pixel loss and gradient loss, which is specifically expressed as:

[0101]

[0102] Among them, I i represents the infrared images in the infrared image training set, I v represents the visible light image in the visible light image training set, and the fused image output by the image fusion network is I f .

[0103] The third aspect of the present invention discloses an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the image processing method described in the first aspect of the present disclosure are implemented.

[0104] Figure 5 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 5 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the electronic device, or an external keyboard, touch pad or mouse, etc.

[0105] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0106] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the image processing method described in the first aspect of the present disclosure are implemented.

[0107] In summary, the present invention proposes a relatively novel data interaction model from the perspective of network channels, combined with the characteristics of Hankel convolution extended by Hankel matrix, inspired by GSVD. Different from the traditional shared parameters between channels, by using Hankel convolution, data from different channels can be flexibly switched to achieve data interaction. Through this interaction between data, the feature extraction level of multiple inputs is improved to overcome the shortcomings of multi-channel network parallel feature extraction methods, thereby achieving more efficient processing performance.

[0108] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Step S1, acquiring an infrared image and a visible light image; Among them, M infrared images and corresponding M visible light images are selected as infrared image training set and visible light image training set respectively; N infrared images and corresponding N visible light images are selected as infrared image test set and visible light image test set respectively; M>N; Step S2, constructing an image processing system; The image processing system includes a data interaction module and an image fusion network. The data interaction module includes an interactive element calculation module and four Hankel convolution modules. The image fusion network is an IFCNN network. The data interaction module and the image fusion network are connected in series. Step S3, training the image processing system using the infrared image training set and the visible light image training set, and testing the trained image processing system using the infrared image test set and the visible light image test set; Among them, the loss function in the training process is calculated using pixel loss and gradient loss.

2. An image processing method according to claim 1, characterized in that: The data interaction module includes two channels, and two Hankel convolution modules are respectively configured in the first channel and the second channel; wherein, during the training / testing process: The data interaction module receives an input infrared grayscale image and a visible light grayscale image; In the first channel, the infrared grayscale image passes through the first Hankel convolution module, the interactive element calculation module and the second Hankel convolution module respectively to obtain infrared image data after interactive processing; In the second channel, the visible light grayscale image passes through the third Hankel convolution module, the interactive element calculation module and the third Hankel convolution module respectively to obtain the visible light image data after interactive processing.

3. An image processing method according to claim 2, characterized in that: P1 and P2 represent the infrared grayscale image and the visible light grayscale image, respectively; The output of the first Hankel convolution module is The output of the third Hankel convolution module is Represents the initialized convolution basis; The interactive element calculation module uses the interactive element B 1,2 To interactively switch the output of the first Hankel convolution module and the output of the third Hankel convolution module, the interactive switching process is described as: The outputs of the interactive element calculation module in the first channel and the second channel are sent to the second Hankel convolution module and the fourth Hankel convolution module respectively.

4. An image processing method according to claim 3, characterized in that: The loss function in the training process is calculated using pixel loss and gradient loss, which is specifically expressed as: Among them, I i represents the infrared images in the infrared image training set, I v represents the visible light image in the visible light image training set, and the fused image output by the image fusion network is I f .

5. An image processing system, characterized in that: The system comprises: a data interaction module and an image fusion network, wherein the data interaction module comprises an interactive element calculation module and four Hankel convolution modules, the image fusion network is an IFCNN network, and the data interaction module and the image fusion network are connected in series; wherein: Acquire infrared images and visible light images; Select M infrared images and corresponding M visible light images as infrared image training set and visible light image training set respectively; select N infrared images and corresponding N visible light images as infrared image test set and visible light image test set respectively; M>N; The image processing system is trained using the infrared image training set and the visible light image training set, and the trained image processing system is tested using the infrared image test set and the visible light image test set; The loss function during training is calculated using pixel loss and gradient loss.

6. An image processing system according to claim 5, characterized in that: The data interaction module includes two channels, and two Hankel convolution modules are respectively configured in the first channel and the second channel; wherein, during the training / testing process: The data interaction module receives an input infrared grayscale image and a visible light grayscale image; In the first channel, the infrared grayscale image passes through the first Hankel convolution module, the interactive element calculation module and the second Hankel convolution module respectively to obtain infrared image data after interactive processing; In the second channel, the visible light grayscale image passes through the third Hankel convolution module, the interactive element calculation module and the third Hankel convolution module respectively to obtain the visible light image data after interactive processing.

7. An image processing system according to claim 6, characterized in that: P1 and P2 represent the infrared grayscale image and the visible light grayscale image, respectively; The output of the first Hankel convolution module is The output of the third Hankel convolution module is Represents the initialized convolution basis; The interactive element calculation module uses the interactive element B 1,2 To interactively switch the output of the first Hankel convolution module and the output of the third Hankel convolution module, the interactive switching process is described as: The outputs of the interactive element calculation module in the first channel and the second channel are sent to the second Hankel convolution module and the fourth Hankel convolution module respectively.

8. An image processing system according to claim 7, characterized in that: The loss function in the training process is calculated using pixel loss and gradient loss, which is specifically expressed as: Among them, I i represents the infrared images in the infrared image training set, I v represents the visible light image in the visible light image training set, and the fused image output by the image fusion network is I f .

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the image processing method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the image processing method according to any one of claims 1 to 4 are implemented.

Images