An image processing method and system

Abstract

The application provides an image processing method and system, and belongs to the technical field of image fusion. The application is inspired by GSVD and combines the characteristics of Hankel convolution extended by a Hankel matrix to propose a novel data interaction model from the perspective of network channels. Different from the traditional sharing parameters between channels, the data of different channels can be flexibly switched by using the Hankel convolution, so that the data interaction is realized. The feature extraction level of multiple inputs is improved through the interaction between the data, so as to overcome the shortcomings of the parallel feature extraction mode of the multi-channel network, and thus the processing efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of image fusion technology, and in particular relates to an image processing method and system. Background Technology

[0002] Deep learning has a wide range of applications and significant implications in multi-input tasks. Multi-input tasks involve processing multiple different or similar input data and then performing various subsequent processing steps to complete different tasks. However, for multi-input processing, existing methods mostly process each input data using a separate channel, and then unify the outputs of each network into a subsequent model to achieve the desired effect. This approach has yielded good results in many tasks, such as CT image reconstruction and multimodal fusion. Compared to traditional methods, deep learning can achieve higher precision processing and has a significant advantage in handling large datasets.

[0003] However, when multiple networks are used for processing in this network configuration, the channels lack close interaction, leading to a lack of comprehensive consideration of multiple input data during feature extraction and reduced data utilization efficiency. Therefore, from the perspective of channel interaction, it is meaningful to study robust channel interaction modules to improve data utilization.

[0004] From the perspective of existing network models, taking the Transformer as an example, a channel attention mechanism is established by using the Transformer to provide appropriate weights for the features of each channel, thereby realizing the connection between channels. Its main components are:

[0005] Self-attention: This is one of the core concepts of Transformer. It allows the model to consider all positions in the input sequence simultaneously, rather than processing them step-by-step like a recurrent neural network (RNN) or convolutional neural network (CNN). Self-attention allows the model to assign different attention weights to different parts of the input sequence, thereby better capturing semantic relationships.

[0006] Multi-head attention: The self-attention mechanism in Transformer is extended to multiple attention heads, each of which can learn different attention weights to better capture different types of relationships. Multi-head attention allows the model to process different information subspaces in parallel.

[0007] Encoder and Decoder: A Transformer typically includes an encoder for processing the input sequence and a decoder for generating the output sequence, making it suitable for sequence-to-sequence tasks such as machine translation.

[0008] The above processing methods cannot effectively establish connections between data during feature extraction, fully utilize the advantages of multiple input data, and the combined application of multiple networks will also increase the difficulty of training network models. Summary of the Invention

[0009] This invention addresses the shortcomings of existing technologies by proposing an image processing solution.

[0010] The first aspect of this invention provides an image processing method, the method comprising:

[0011] Step S1: Acquire infrared and visible light images;

[0012] Specifically, M infrared images and M corresponding visible light images are selected as the infrared image training set and the visible light image training set, respectively; N infrared images and N corresponding visible light images are selected as the infrared image test set and the visible light image test set, respectively; M > N;

[0013] Step S2: Construct an image processing system;

[0014] The image processing system includes a data interaction module and an image fusion network. The data interaction module includes an interaction meta-computation module and four Hankel convolution modules. The image fusion network is an IFCNN network, and the data interaction module and the image fusion network are connected in series.

[0015] Step S3: Train the image processing system using the infrared image training set and the visible light image training set, and test the trained image processing system using the infrared image test set and the visible light image test set.

[0016] The loss function during training is calculated using pixel loss and gradient loss.

[0017] According to the method of the first aspect, the data interaction module includes two channels, with two Hankel convolutional modules configured in the first and second channels respectively; wherein, during training / testing:

[0018] The data interaction module receives input infrared grayscale images and visible light grayscale images;

[0019] In the first channel, the infrared grayscale image is processed by the first Hankel convolution module, the interactive element calculation module, and the second Hankel convolution module to obtain interactively processed infrared image data.

[0020] In the second channel, the visible light grayscale image is processed by the third Hankel convolution module, the interactive meta-computation module, and the third Hankel convolution module to obtain interactively processed visible light image data.

[0021] According to the method in the first aspect, P1 and P2 represent the infrared grayscale image and the visible light grayscale image, respectively; the output of the first Hankel convolutional module is... The output of the third Hankel convolutional module is This represents the initial convolutional basis; the interaction meta-computation module utilizes the interaction meta-B. 1,2 The outputs of the first and third Hankel convolutional modules are interactively switched. The interactive switching process is described as follows: The outputs of the interactive meta-computation module in the first and second channels are sent to the second and fourth Hankel convolution modules, respectively.

[0022] Based on the method in the first aspect, the loss function during training is calculated using pixel loss and gradient loss, specifically expressed as:

[0023]

[0024] Among them, I i I represents the infrared images in the infrared image training set. v This represents the visible light images in the visible light image training set. The fused image output by the image fusion network is I. f .

[0025] A second aspect of this invention provides an image processing system, comprising: a data interaction module and an image fusion network. The data interaction module includes an interaction meta-computation module and four Hankel convolutional modules. The image fusion network is an IFCNN network, and the data interaction module and the image fusion network are connected in series.

[0026] Acquire infrared and visible light images;

[0027] Select M infrared images and M corresponding visible light images as the infrared image training set and the visible light image training set, respectively; select N infrared images and N corresponding visible light images as the infrared image test set and the visible light image test set, respectively; M > N;

[0028] The image processing system was trained using infrared and visible light image training sets, and tested using infrared and visible light image test sets.

[0029] The loss function during training is calculated using pixel loss and gradient loss.

[0030] According to the system in the second aspect, the data interaction module includes two channels, with two Hankel convolutional modules configured in the first and second channels respectively; wherein, during training / testing:

[0031] The data interaction module receives input infrared grayscale images and visible light grayscale images;

[0032] In the first channel, the infrared grayscale image is processed by the first Hankel convolution module, the interactive element calculation module, and the second Hankel convolution module to obtain interactively processed infrared image data.

[0033] In the second channel, the visible light grayscale image is processed by the third Hankel convolution module, the interactive meta-computation module, and the third Hankel convolution module to obtain interactively processed visible light image data.

[0034] According to the system in the second aspect, P1 and P2 represent the infrared grayscale image and the visible light grayscale image, respectively; the output of the first Hankel convolution module is... The output of the third Hankel convolutional module is This represents the initial convolutional basis; the interaction meta-computation module utilizes the interaction meta-B. 1,2 The outputs of the first and third Hankel convolutional modules are interactively switched. The interactive switching process is described as follows: The outputs of the interactive meta-computation module in the first and second channels are sent to the second and fourth Hankel convolution modules, respectively.

[0035] According to the system in the second aspect, the loss function during training is calculated using pixel loss and gradient loss, specifically expressed as:

[0036]

[0037] Among them, I i I represents the infrared images in the infrared image training set. v This represents the visible light images in the visible light image training set. The fused image output by the image fusion network is I. f .

[0038] A third aspect of this invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the image processing method described in the first aspect of this disclosure.

[0039] A fourth aspect of this invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image processing method described in the first aspect of this disclosure.

[0040] In summary, this invention, starting from the perspective of network channels and combining the characteristics of Hankel convolution extended by the Hankel matrix, and inspired by GSVD, proposes a relatively novel data interaction model. Unlike traditional channel-to-channel parameter sharing, by using Hankel convolution, data from different channels can be flexibly switched, thereby achieving data interaction. This data interaction improves the feature extraction level of multiple inputs, overcoming the shortcomings of multi-path network parallel feature extraction methods, thus achieving more efficient processing performance. Attached Figure Description

[0041] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the structure of a data processing system according to an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of a data interaction module according to an embodiment of the present invention.

[0044] Figure 3 This is a schematic diagram of the interactive switching process according to an embodiment of the present invention.

[0045] Figure 4 This is a schematic diagram of image fusion according to an embodiment of the present invention.

[0046] Figure 5 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] The first aspect of this invention provides an image processing method, the method comprising (e.g.) Figure 1 As shown):

[0049] Step S1: Acquire infrared and visible light images;

[0050] Specifically, M infrared images and M corresponding visible light images are selected as the infrared image training set and the visible light image training set, respectively; N infrared images and N corresponding visible light images are selected as the infrared image test set and the visible light image test set, respectively; M > N;

[0051] Step S2: Construct an image processing system;

[0052] The image processing system includes a data interaction module and an image fusion network. The data interaction module includes an interaction meta-computation module and four Hankel convolution modules. The image fusion network is an IFCNN network, and the data interaction module and the image fusion network are connected in series.

[0053] Step S3: Train the image processing system using the infrared image training set and the visible light image training set, and test the trained image processing system using the infrared image test set and the visible light image test set.

[0054] The loss function during training is calculated using pixel loss and gradient loss.

[0055] According to the method of the first aspect, the data interaction module includes two channels, with two Hankel convolutional modules configured in the first and second channels respectively; wherein, during training / testing:

[0056] The data interaction module receives input infrared grayscale images and visible light grayscale images;

[0057] In the first channel, the infrared grayscale image is processed by the first Hankel convolution module, the interactive element calculation module, and the second Hankel convolution module to obtain interactively processed infrared image data.

[0058] In the second channel, the visible light grayscale image is processed by the third Hankel convolution module, the interactive meta-computation module, and the third Hankel convolution module to obtain interactively processed visible light image data.

[0059] According to the method in the first aspect, P1 and P2 represent the infrared grayscale image and the visible light grayscale image, respectively; the output of the first Hankel convolutional module is... The output of the third Hankel convolutional module is This represents the initial convolutional basis; the interaction meta-computation module utilizes the interaction meta-B.1,2 The outputs of the first and third Hankel convolutional modules are interactively switched. The interactive switching process is described as follows: The outputs of the interactive meta-computation module in the first and second channels are sent to the second and fourth Hankel convolution modules, respectively.

[0060] Based on the method in the first aspect, the loss function during training is calculated using pixel loss and gradient loss, specifically expressed as:

[0061]

[0062] Among them, I i I represents the infrared images in the infrared image training set. v This represents the visible light images in the visible light image training set. The fused image output by the image fusion network is I. f .

[0063] In some embodiments, the proposed network interaction module enables data interaction and transmission between two channels. The data interaction module mainly consists of two parts:

[0064] (1) Interactive meta-computation process (including interaction and switching), such as Figure 2-3 As shown in the figure, the Combine part is constructed by two input data to realize the switching and interaction between network input data; (2) The network propagation process mainly includes two convolutions. The first convolution is a regular Hankel convolution, and the second convolution is achieved by convolution with the interaction element to realize the interaction of data.

[0065] In addition, the data interaction module can generally be connected in series with other models, and subsequent processing can still be divided into two networks for separate propagation.

[0066] In some embodiments, the interaction module provides a solution for establishing connections between networks. Taking two inputs as an example, the basic definition of Hankel convolution is given first:

[0067] Input P∈R nl×p and convolution basis Ψ∈R p×q Each column of the input can be viewed as l n-dimensional vectors:

[0068]

[0069] Hankel convolution can be represented as:

[0070]

[0071] in, This indicates that the matrix is ​​flipped column by column. for:

[0072]

[0073] The interaction element is computed using two inputs, P1 and P2. First, the input, partitioned into column vectors, is transposed. However, unlike the general transpose method, each smaller vector is transposed separately during computation.

[0074]

[0075] Then the transposed result and Find the inverse, and you will get and satisfy When a matrix does not have an inverse, a generalized inverse can be used as a substitute, or the image can be segmented, input into the interaction module separately, and then the output can be reassembled. The interaction element is calculated as follows:

[0076]

[0077] Initialize the convolution basis Ψ and perform operations on the two inputs respectively, as follows:

[0078]

[0079] Interactive elements facilitate data switching and interaction. These elements can switch input data that plays a role in a network, utilizing the interaction element B. 1,2 B 1,2 When switching occurs, it can be represented as:

[0080]

[0081] After obtaining the output of the completed interaction, the switched data will be processed according to the task requirements.

[0082] The interactive module proposed in this invention can effectively establish bridges between data points, thereby enabling interaction between network channels and improving the performance of multi-input networks. This module is highly flexible and can be added to the network front-end, middle, and back-end, depending on the specific circumstances.

[0083] In some embodiments, to verify the effectiveness of the data interaction method provided by the proposed interaction module, a multimodal image (infrared and visible light) fusion experiment was conducted. The classic image fusion network IFCNN was used as the base network, and the interaction module and the IFCNN model were connected in series. Furthermore, to more intuitively demonstrate the gain of infrared and visible light fusion, instead of using multi-channel preprocessing as in the original model, both images were directly converted to grayscale before being input into the network. Specific parameters are shown in Table 1.

[0084] Table 1: Main Technical Specifications Used

[0085]

[0086] After registration with FILR, 1000 images were selected as the training set, and the TNO dataset was chosen as the test set. Pixel loss and gradient loss were selected as the loss functions, and the visible light images were represented by I0. v The infrared image is I i The output fused image is I f The loss function can be expressed as:

[0087]

[0088] The fusion result is as follows Figure 4 As shown, observation reveals that networks with interactive modules exhibit significant advantages in detail restoration. Compared to using IFCNN alone, interactive modules can preserve clearer textures while maintaining fusion effectiveness, thus improving subsequent detection and recognition accuracy. However, extensive experiments have shown that while interactive modules still offer advantages in detail restoration when inputs exhibit substantial differences, they may have a negative effect on certain image patches. Therefore, preprocessing is necessary for some images to reduce inter-image differences and improve network performance.

[0089] In summary, this invention provides a novel interaction method for multi-input networks, establishing a bridge between channels to comprehensively utilize the features of multiple sets of data. Combining the proposed interaction module with the network can effectively improve network performance, establish connections between channels, and thus find common features in high-dimensional space. The interaction structure proposed in this technology is more flexible in application, allowing for complete interaction at different locations in the network, thereby addressing more diverse task requirements.

[0090] A second aspect of this invention provides an image processing system, comprising: a data interaction module and an image fusion network. The data interaction module includes an interaction meta-computation module and four Hankel convolutional modules. The image fusion network is an IFCNN network, and the data interaction module and the image fusion network are connected in series.

[0091] Acquire infrared and visible light images;

[0092] Select M infrared images and M corresponding visible light images as the infrared image training set and the visible light image training set, respectively; select N infrared images and N corresponding visible light images as the infrared image test set and the visible light image test set, respectively; M > N;

[0093] The image processing system was trained using infrared and visible light image training sets, and tested using infrared and visible light image test sets.

[0094] The loss function during training is calculated using pixel loss and gradient loss.

[0095] According to the system in the second aspect, the data interaction module includes two channels, with two Hankel convolutional modules configured in the first and second channels respectively; wherein, during training / testing:

[0096] The data interaction module receives input infrared grayscale images and visible light grayscale images;

[0097] In the first channel, the infrared grayscale image is processed by the first Hankel convolution module, the interactive element calculation module, and the second Hankel convolution module to obtain interactively processed infrared image data.

[0098] In the second channel, the visible light grayscale image is processed by the third Hankel convolution module, the interactive meta-computation module, and the third Hankel convolution module to obtain interactively processed visible light image data.

[0099] According to the system in the second aspect, P1 and P2 represent the infrared grayscale image and the visible light grayscale image, respectively; the output of the first Hankel convolution module is... The output of the third Hankel convolutional module is This represents the initial convolutional basis; the interaction meta-computation module utilizes the interaction meta-B. 1,2 The outputs of the first and third Hankel convolutional modules are interactively switched. The interactive switching process is described as follows: The outputs of the interactive meta-computation module in the first and second channels are sent to the second and fourth Hankel convolution modules, respectively.

[0100] According to the system in the second aspect, the loss function during training is calculated using pixel loss and gradient loss, specifically expressed as:

[0101]

[0102] Among them, I i I represents the infrared images in the infrared image training set. v This represents the visible light images in the visible light image training set. The fused image output by the image fusion network is I. f .

[0103] A third aspect of this invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the image processing method described in the first aspect of this disclosure.

[0104] Figure 5 This is a structural diagram of an electronic device according to an embodiment of the present invention, such as... Figure 5 As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, Near Field Communication (NFC), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0105] Those skilled in the art will understand that Figure 5 The structure shown is merely a structural diagram of the part related to the technical solution of this disclosure and does not constitute a limitation on the electronic device to which the solution of this application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0106] A fourth aspect of this invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image processing method described in the first aspect of this disclosure.

[0107] In summary, this invention, starting from the perspective of network channels and combining the characteristics of Hankel convolution extended by the Hankel matrix, and inspired by GSVD, proposes a relatively novel data interaction model. Unlike traditional channel-to-channel parameter sharing, by using Hankel convolution, data from different channels can be flexibly switched, thereby achieving data interaction. This data interaction improves the feature extraction level of multiple inputs, overcoming the shortcomings of multi-path network parallel feature extraction methods, thus achieving more efficient processing performance.

[0108] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Step S1: Acquire infrared and visible light images; Specifically, M infrared images and M corresponding visible light images are selected as the infrared image training set and the visible light image training set, respectively; N infrared images and N corresponding visible light images are selected as the infrared image test set and the visible light image test set, respectively; M > N; Step S2: Construct an image processing system; The image processing system includes a data interaction module and an image fusion network. The data interaction module includes an interaction meta-computation module and four Hankel convolution modules. The image fusion network is an IFCNN network, and the data interaction module and the image fusion network are connected in series. Step S3: Train the image processing system using the infrared image training set and the visible light image training set, and test the trained image processing system using the infrared image test set and the visible light image test set. Among them, the loss function during the training process is calculated using pixel loss and gradient loss; The data interaction module includes two channels, with two Hankel convolutional modules configured in the first and second channels respectively; during training / testing: The data interaction module receives input infrared grayscale images and visible light grayscale images; In the first channel, the infrared grayscale image is processed by the first Hankel convolution module, the interactive element calculation module, and the second Hankel convolution module to obtain interactively processed infrared image data. In the second channel, the visible light grayscale image is processed by the third Hankel convolution module, the interactive meta-computation module, and the third Hankel convolution module to obtain the interactively processed visible light image data. in, and Let the infrared grayscale image and the visible light grayscale image be represented respectively; then: The output of the first Hankel convolutional module is The output of the third Hankel convolutional module is , Indicates the initial convolutional basis; The interactive meta-computation module utilizes interactive meta-computation. The outputs of the first and third Hankel convolutional modules are interactively switched. The interactive switching process is described as follows: ; The outputs of the interactive meta-computation module in the first and second channels are sent to the second Hankel convolution module and the fourth Hankel convolution module, respectively. Among them, interactive elements The calculation method is as follows: in, , where n represents the dimension.

2. The image processing method according to claim 1, characterized in that, The loss function during training is calculated using pixel loss and gradient loss, and is specifically expressed as follows: in, This represents the infrared images in the infrared image training set. This represents the visible light images in the visible light image training set. The fused image output by the image fusion network is... .

3. An image processing system, characterized in that, The system includes: a data interaction module and an image fusion network. The data interaction module includes one interaction meta-computation module and four Hankel convolutional modules. The image fusion network is an IFCNN network, and the data interaction module and the image fusion network are connected in series. Acquire infrared and visible light images; Select M infrared images and M corresponding visible light images as the infrared image training set and the visible light image training set, respectively; select N infrared images and N corresponding visible light images as the infrared image test set and the visible light image test set, respectively; M > N; The image processing system was trained using infrared and visible light image training sets, and tested using infrared and visible light image test sets. The loss function during training is calculated using pixel loss and gradient loss. The data interaction module includes two channels, with two Hankel convolutional modules configured in the first and second channels respectively; wherein, during training / testing: The data interaction module receives input infrared grayscale images and visible light grayscale images; In the first channel, the infrared grayscale image is processed by the first Hankel convolution module, the interactive element calculation module, and the second Hankel convolution module to obtain interactively processed infrared image data. In the second channel, the visible light grayscale image is processed by the third Hankel convolution module, the interactive meta-computation module, and the third Hankel convolution module to obtain the interactively processed visible light image data. and Let the infrared grayscale image and the visible light grayscale image be represented respectively; then: The output of the first Hankel convolutional module is The output of the third Hankel convolutional module is , Indicates the initial convolutional basis; The interactive meta-computation module utilizes interactive meta-computation. The outputs of the first and third Hankel convolutional modules are interactively switched. The interactive switching process is described as follows: ; The outputs of the interactive meta-computation module in the first and second channels are sent to the second Hankel convolution module and the fourth Hankel convolution module, respectively. Among them, interactive elements The calculation method is as follows: in, , where n represents the dimension.

4. The image processing system according to claim 3, characterized in that, The loss function during training is calculated using pixel loss and gradient loss, and is specifically expressed as follows: in, This represents the infrared images in the infrared image training set. This represents the visible light images in the visible light image training set. The fused image output by the image fusion network is... .

5. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the image processing method according to claim 1 or 2.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image processing method according to claim 1 or 2.

Images