Image processing method and system for multi-source data

Abstract

The invention provides an image processing method and system for multi-source data, and belongs to the technical field of image processing. From the perspective of multi-data interaction and light weight, a new convolutional neural network model is designed for image processing, such as noise reduction. The model is mainly a group of encoder-decoder structures, data interaction is completed by ingeniously applying convolution in the encoding and decoding processes, the relation between input data is established, and the advantages of multiple data are fully utilized. In addition, the model is simple in structure, and can have the processing capability of a classical network only through several layers of network structures on the premise of reasonably applying multiple data.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image processing method and system for multi-source data. Background Art

[0002] Deep learning is a hot research direction at present. With the continuous development of this technology, it has played a very important role in many fields. Among them, convolutional neural network (CNN) is one of the most important methods in deep learning. Its excellent feature extraction ability has achieved good results in quite a few tasks. In the field of processing data provided by distributed platforms, CNN provides model accuracy and robustness superior to traditional processing methods, and has become a widely used method in this field. However, most of the existing models are applied research of classical networks, using a single input network to extract the features of multiple data separately and then perform subsequent processing. This method does not consider the inherent correlation and difference between multiple data, and does not fully play the advantages of multiple data. Although blindly deepening the network model will improve the efficiency of the network, it will increase the complexity of the model and increase the difficulty of optimization. Therefore, designing a more reasonable distributed data processing network, taking into account the needs of network performance and network lightweight, to improve the usability of the network has become an urgent problem to be solved.

[0003] From the perspective of existing network models, the technology of extracting features of multiple data respectively through multiple parallel network modules, and then connecting them with feature processing modules to realize the processing of multiple data is similar to this patent. It is mainly divided into two parts: first, a group of networks are designed to extract features of the data respectively; second, the output of the feature extraction module is uniformly input into the subsequent feature processing module, which can be a network or a traditional optimization model. This processing method cannot effectively establish the connection between data in the process of extracting features, and fully utilize the advantages of multiple input data. In addition, the combined application of multiple networks will also increase the difficulty of network model training. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention proposes an image processing solution for multi-source data.

[0005] A first aspect of the present invention provides an image processing method for multi-source data, the method comprising:

[0006] Step S1, acquiring a first image and a second image as multi-source data;

[0007] Step S2, constructing and training an image processing system;

[0008] Step S3: Process the multi-source data using the trained image processing system.

[0009] in:

[0010] The image processing system includes a precoding module, a coding module and a decoding module; the precoding module includes a precoding convolution Pconv, an expansion module Ex, and an interactive computing module Com; the coding module includes a plurality of coding units, each of which includes a coding interactive convolution Conv and an expansion module Ex; the decoding module includes a plurality of decoding units, an inverse expansion module Dex independent of the plurality of decoding units, and a deprecoding convolution Dpconv, each of which includes a decoding interactive convolution Dconv and an inverse expansion module Dex;

[0011] The interactive element output by the interactive calculation module Com is fed to each encoding unit included in the encoding module to perform encoding interactive convolution calculation; at the same time, the interactive element is also fed to each decoding unit included in the decoding module to perform decoding interactive convolution calculation;

[0012] The output of each decoding unit included in the decoding module is fed to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is fed to the de-precoding convolution Dpconv.

[0013] According to the method of the first aspect, in the precoding stage, the precoding module is called to execute:

[0014] receiving a first image and a second image as multi-source data;

[0015] The multi-source data is pre-encoded using the pre-coding convolution Pconv, and the pre-coding result is input into the extension module Ex. The output of the extension module Ex is used as the pre-coding result, which is expressed as:

[0016]

[0017] Among them, C Pi represents the precoding result, P i Represents multi-source data, Represents the initialized convolution basis;

[0018] At the same time, the interactive calculation module Com calculates the interactive element B based on the first image and the second image i,j , interaction element B i,j It is then fed to the encoding module and the decoding module; the interaction element B i,j It is expressed as:

[0019]

[0020] in, Represents the transposed form, l represents the number of vector rows, p represents the number of vector columns, and n represents the number of channels.

[0021] According to the method of the first aspect, in the encoding stage, the encoding module is called to execute:

[0022] Receiving a precoding result, the precoding result is input into a coding unit, each coding unit is connected in series, the output of the previous coding unit is used as the input of the next coding unit, and the output of the last coding unit is used as the coding result of the coding module;

[0023] For each coding unit, the encoding process is expressed as:

[0024]

[0025] Among them, C C' Represents the input of the current coding unit, C C represents the encoding result of the current coding unit, ρ represents the activation function, and Φ represents the expansion basis.

[0026] According to the method of the first aspect, in the decoding stage, the decoding module is called to execute:

[0027] Receiving the encoding result, the encoding result is input to the decoding unit, each decoding unit is connected in series, and the output of the previous decoding unit is used as the input of the next decoding unit;

[0028] The output of each decoding unit is input to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is input to the de-precoding convolution Dpconv, and the output of the de-precoding convolution Dpconv and the output of the last encoding unit are input to the de-precoding convolution Dpconv;

[0029] For each decoding unit, the decoding process is expressed as:

[0030]

[0031] in, Represents the input of the current decoding unit, Indicates the decoding result of the current decoding unit;

[0032] The output of the deprecoded convolution Dpconv is expressed as:

[0033]

[0034] Among them, P i and P j Respectively represent the results of the first image and the second image being processed by the image processing system, represents the output of the last encoding unit, Represents the vectorized result after performing dual basis on the convolution basis.

[0035] A second aspect of the present invention provides an image processing system for multi-source data, the system is used to receive a first image and a second image as multi-source data, and use a trained image processing system to process the multi-source data; the system includes: a pre-coding module, an encoding module and a decoding module; wherein:

[0036] The precoding module includes a precoding convolution Pconv, an expansion module Ex, and an interactive calculation module Com; the encoding module includes a number of encoding units, each of which includes an encoding interactive convolution Conv and an expansion module Ex; the decoding module includes a number of decoding units, an inverse expansion module Dex independent of the number of decoding units, and a deprecoding convolution Dpconv, each of which includes a decoding interactive convolution Dconv and an inverse expansion module Dex; wherein:

[0037] The interactive element output by the interactive calculation module Com is fed to each encoding unit included in the encoding module to perform encoding interactive convolution calculation; at the same time, the interactive element is also fed to each decoding unit included in the decoding module to perform decoding interactive convolution calculation;

[0038] The output of each decoding unit included in the decoding module is fed to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is fed to the de-precoding convolution Dpconv.

[0039] According to the system of the second aspect, in the precoding stage, the precoding module is called to execute:

[0040] receiving a first image and a second image as multi-source data;

[0041] The multi-source data is pre-encoded using the pre-coding convolution Pconv, and the pre-coding result is input into the extension module Ex. The output of the extension module Ex is used as the pre-coding result, which is expressed as:

[0042]

[0043] Among them, C Pi represents the precoding result, P i Represents multi-source data, Represents the initialized convolution basis;

[0044] At the same time, the interactive calculation module Com calculates the interactive element B based on the first image and the second image i,j , interaction element B i,j It is then fed to the encoding module and the decoding module; the interaction element B i,j It is expressed as:

[0045]

[0046] in, Represents the transposed form, l represents the number of vector rows, p represents the number of vector columns, and n represents the number of channels.

[0047] According to the system of the second aspect, in the encoding stage, the encoding module is called to execute:

[0048] Receiving a precoding result, the precoding result is input into a coding unit, each coding unit is connected in series, the output of the previous coding unit is used as the input of the next coding unit, and the output of the last coding unit is used as the coding result of the coding module;

[0049] For each coding unit, the encoding process is expressed as:

[0050]

[0051] Among them, C C' Represents the input of the current coding unit, C C represents the encoding result of the current coding unit, ρ represents the activation function, and Φ represents the expansion basis.

[0052] According to the system of the second aspect, in the decoding stage, the decoding module is called to execute:

[0053] Receiving the encoding result, the encoding result is input to the decoding unit, each decoding unit is connected in series, and the output of the previous decoding unit is used as the input of the next decoding unit;

[0054] The output of each decoding unit is input to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is input to the de-precoding convolution Dpconv, and the output of the de-precoding convolution Dpconv and the output of the last encoding unit are input to the de-precoding convolution Dpconv;

[0055] For each decoding unit, the decoding process is expressed as:

[0056]

[0057] in, Represents the input of the current decoding unit, Indicates the decoding result of the current decoding unit;

[0058] The output of the deprecoded convolution Dpconv is expressed as:

[0059]

[0060] Among them, P i and P jRespectively represent the results of the first image and the second image being processed by the image processing system, represents the output of the last encoding unit, Represents the vectorized result after performing dual basis on the convolution basis.

[0061] The third aspect of the present invention discloses an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the image processing method for multi-source data described in the first aspect of the present disclosure are implemented.

[0062] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the image processing method for multi-source data described in the first aspect of the present disclosure are implemented.

[0063] In summary, the present invention designs a new convolutional neural network model for image processing from the perspective of multi-data interaction and lightweight. The model is mainly a set of encoder-decoder structures, which cleverly use convolution to complete data interaction during encoding and decoding, establish connections between input data, and make full use of the advantages of multiple data. In addition, the model has a simple structure. Under the premise of reasonable use of multiple data, only a few layers of network structure are needed to have the processing power of the classic network. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 Schematic diagram of the structure of a data processing system according to an embodiment of the present invention.

[0066] Figure 2 FIG. 4 is a schematic diagram of image noise reduction processing according to an embodiment of the present invention.

[0067] Figure 3 Schematic diagram of image stitching processing according to an embodiment of the present invention.

[0068] Figure 4 The figure is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0070] A first aspect of the present invention provides an image processing method for multi-source data, the method comprising:

[0071] Step S1, acquiring a first image and a second image as multi-source data;

[0072] Step S2, constructing and training an image processing system;

[0073] Step S3: Process the multi-source data using the trained image processing system.

[0074] in:

[0075] Image processing systems (such as Figure 1 The invention comprises a precoding module, a coding module and a decoding module; the precoding module comprises a precoding convolution Pconv, an expansion module Ex, and an interactive calculation module Com; the coding module comprises a plurality of coding units, each of which comprises a coding interactive convolution Conv and an expansion module Ex; the decoding module comprises a plurality of decoding units, an inverse expansion module Dex independent of the plurality of decoding units, and a deprecoding convolution Dpconv, each of which comprises a decoding interactive convolution Dconv and an inverse expansion module Dex;

[0076] The interactive element output by the interactive calculation module Com is fed to each encoding unit included in the encoding module to perform encoding interactive convolution calculation; at the same time, the interactive element is also fed to each decoding unit included in the decoding module to perform decoding interactive convolution calculation;

[0077] The output of each decoding unit included in the decoding module is fed to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is fed to the de-precoding convolution Dpconv.

[0078] According to the method of the first aspect, in the precoding stage, the precoding module is called to execute:

[0079] receiving a first image and a second image as multi-source data;

[0080] The multi-source data is pre-encoded using the pre-coding convolution Pconv, and the pre-coding result is input into the extension module Ex. The output of the extension module Ex is used as the pre-coding result, which is expressed as:

[0081]

[0082] Among them, C Pi represents the precoding result, P i Represents multi-source data, Represents the initialized convolution basis;

[0083] At the same time, the interactive calculation module Com calculates the interactive element B based on the first image and the second image i,j , interaction element B i,j It is then fed to the encoding module and the decoding module; the interaction element B i,j It is expressed as:

[0084]

[0085] in, Represents the transposed form, l represents the number of vector rows, p represents the number of vector columns, and n represents the number of channels.

[0086] According to the method of the first aspect, in the encoding stage, the encoding module is called to execute:

[0087] Receiving a precoding result, the precoding result is input into a coding unit, each coding unit is connected in series, the output of the previous coding unit is used as the input of the next coding unit, and the output of the last coding unit is used as the coding result of the coding module;

[0088] For each coding unit, the encoding process is expressed as:

[0089]

[0090] Among them, C C' Represents the input of the current coding unit, C C represents the encoding result of the current coding unit, ρ represents the activation function, and Φ represents the expansion basis.

[0091] According to the method of the first aspect, in the decoding stage, the decoding module is called to execute:

[0092] Receiving the encoding result, the encoding result is input to the decoding unit, each decoding unit is connected in series, and the output of the previous decoding unit is used as the input of the next decoding unit;

[0093] The output of each decoding unit is input to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is input to the de-precoding convolution Dpconv, and the output of the de-precoding convolution Dpconv and the output of the last encoding unit are input to the de-precoding convolution Dpconv;

[0094] For each decoding unit, the decoding process is expressed as:

[0095]

[0096] in, Represents the input of the current decoding unit, Indicates the decoding result of the current decoding unit;

[0097] The output of the deprecoded convolution Dpconv is expressed as:

[0098]

[0099] Among them, P i and P j Respectively represent the results of the first image and the second image being processed by the image processing system, represents the output of the last encoding unit, Represents the vectorized result after performing dual basis on the convolution basis.

[0100] Example 1

[0101] Take two inputs as an example, Figure 1 As shown in the figure, the interactive convolution was pioneered based on Hankel convolution, and a network was constructed based on this, which can be divided into three parts: precoding: first precoding the input data, which is the basis of the interactive convolution in the subsequent encoder; encoder: through interactive convolution, the interaction between data is completed in the encoding process; decoder: still uses the form of interactive convolution, and realizes the decoding process by selecting the dual basis in the encoder.

[0102] The convolution method used in this model is Hankel convolution, which takes the input P∈R nl×p And the convolution basis Ψ∈R p×q , each column of the input is considered as a vector of n dimensions:

[0103]

[0104] Hankel convolution can be expressed as:

[0105]

[0106] in, Indicates that the matrix is ​​flipped column by column. for:

[0107]

[0108] Pre-convolve the input and calculate the interaction element. Initialize the convolution basis Ψ and select an input for Hankel convolution. The specific form is:

[0109]

[0110] Then input the pairing calculation interaction element B i,j , calculated as:

[0111]

[0112] in, is a special transposed form of the input, which can be expressed as:

[0113]

[0114] In order to satisfy the uniqueness of the above inversion, the input data needs to be vectorized before input so that the input dimension satisfies l ≥ np. Then the interaction element and the pre-convolution result are input into the encoder for the next operation.

[0115] The output of the pre-convolution is subjected to interactive convolution, and the convolution basis is expanded to extract deeper features. Interactive convolution is the Hankel convolution of the interactive element and the output of the previous convolution, which can be expressed as Through interactive convolution, the model can switch the active input data of this layer, and can flexibly design the number of interactive convolutions and the participating interactive elements according to the number of inputs and design requirements. After each interactive convolution, it directly interacts with the expansion element Φ i After the operation, select the appropriate activation function for operation.

[0116] Taking input 1 and input 2 as examples, the output of the pre-convolution is C1, the activation function is expressed as ρ(·), and a complete interactive convolution and convolution expansion can be expressed as:

[0117]

[0118] Decode the output of the encoder. Corresponding to the encoder, decoding also includes two parts: deconvolution and the inverse process of expansion.

[0119] First, find the extended base and convolutional basis Satisfy the conditions After the inverse transformation of the expansion module, the interactive convolution is performed, and the convolution result is operated by selecting a suitable activation function. The output of the previous layer is Taking input 1 and input 2 as an example, the input of the previous layer is P1, and the above process can be expressed as:

[0120]

[0121] After several interactive convolutions and inverse transformations of the expansion module, the decoded output is finally obtained through deconvolution, and the dimension is consistent with the input.

[0122] And locally vectorize the dual basis of the convolution basis to obtain Specifically expressed as:

[0123]

[0124] The entire deconvolution is expressed as:

[0125]

[0126] Among them, other inputs can be obtained through interactive convolution:

[0127]

[0128] For ease of understanding, taking two inputs as an example, an interaction is performed during the encoding and decoding process respectively. The whole process can be expressed as:

[0129]

[0130] Example 2

[0131] In order to verify the superiority of the proposed network model, two similar images after preprocessing are used to conduct denoising experiments to complete the verification (the purpose of preprocessing is to improve the low rank of input data and optimize the model processing capability. Here, only data splicing is used as an example). Figure 2 shown.

[0132] Preprocessing: divide the obtained input images with the same level of noise into two groups, cut them into 300×5 image blocks respectively, and obtain two low-rank outputs after splicing, such as Figure 3 shown.

[0133] The preprocessed training set images are input into the model for training. The goal is to obtain clean images without adding noise. The loss function is the two-norm loss, which is specifically expressed as:

[0134]

[0135] Where m, n are the sizes of the image, (i, j) are the corresponding positions, and the trained model is obtained through training.

[0136] After preprocessing, the noisy images in the test set are input into the trained model to obtain the denoised image blocks. In order to avoid the performance advantage caused by the noise statistics, one of the denoised images is selected and spliced ​​to the original image size to obtain the denoised image.

[0137] Programming was completed on pycharm software using pytorch language, and a portable notebook equipped with a 4060 graphics card was used. For training, 10 images of 2040×1360 in the DIV2K dataset were selected, and the same level of simulated noise was added to generate the training set. The test set selected two grayscale image test sets, Set12 and BSD68, and also added simulated noise as input. In order to meet the input size, when processing the Set12 dataset, it is agreed to modify the image size to 512×512. If the input size is not enough, the repeated part will be selected to supplement. The network training parameters are shown in Table 1.

[0138] Table 1: Main technical parameters used

[0139]

[0140] DnCNN, BM3D, Restormer and BRDNet models are selected as comparison to highlight the effect of the proposed network. Due to the small number of training sets, Restormer performs poorly, so the model trained by the original author is used as the comparison result.

[0141] Table 2: Comprehensive denoising experimental results

[0142]

[0143] The experimental results are shown in Table 2. By comparison, it can be found that the proposed model has a significant improvement over the existing methods, giving full play to the advantages of multiple data and effectively improving the processing performance. In addition, the model takes less time to process images, and it only takes less than 0.18 seconds to process an image of size 512×512, which has good processing efficiency.

[0144] Example 3

[0145] In order to further prove the generalization of the proposed model, we also take two inputs as an example and use two preprocessed images with obvious differences for experimental verification. The process is as follows:

[0146] Preprocessing: The obtained same input images with different levels of noise added are divided into two groups, which are cut into 300×5 image blocks and spliced ​​to obtain two low-rank outputs.

[0147] The preprocessed training set images are input into the model for training. The goal is to obtain a clean image without adding noise. The two-norm loss is also used as the loss function to obtain a trained model.

[0148] After preprocessing, the images added to the test set are input into the trained model to obtain the denoised image blocks. In order to avoid the performance advantage caused by the noise statistics, one of the denoised images is selected and spliced ​​to the original image size to obtain the denoised image.

[0149] The experimental equipment is the same as that in Example 1. For training, 10 images of 2040×1360 in the DIV2K dataset were selected, and two different levels of simulated noise were added to generate the training set. The test set selected two grayscale image test sets, Set12 and BSD68, and also added simulated noise as input. In order to meet the input size, the image size was modified to 512×512 when processing the Set12 dataset. If the input size is not enough, the repeated part will be selected to supplement. The network training parameters are shown in Table 3.

[0150] Table 3: Main technical parameters used

[0151]

[0152] For the denoising experiments with different input noise levels, the noise levels for training and testing were set to σ=15 and σ=25 respectively. The results are shown in Table 4.

[0153] Table 4: Complementary denoising experimental results

[0154]

[0155] Through experiments, we can find that when inputting the same image with different levels of background noise, the model can still maintain a high processing effect. Although it is slightly lower than the result of the same level of noise in Example 1, it still shows extremely strong generalization.

[0156] It can be seen that the present invention provides an interactive bridge for multiple input data, making fuller use of the advantages of multiple data; uses a set of parameters to extract the features of multiple data, reduces the complexity of the model, and improves the image processing efficiency; uses reasonable low-rank preprocessing methods, and the processing effect of the model is higher than the capabilities of some existing models; model training is more intuitive, and there is no need to learn specific noise to obtain denoised images, and image reconstruction can be completed directly; the model has better generalization, and can effectively utilize complementary features when processing differential data to maintain good accuracy.

[0157] A second aspect of the present invention provides an image processing system for multi-source data, the system is used to receive a first image and a second image as multi-source data, and use a trained image processing system to process the multi-source data; the system includes: a pre-coding module, an encoding module and a decoding module; wherein:

[0158] The precoding module includes a precoding convolution Pconv, an expansion module Ex, and an interactive calculation module Com; the encoding module includes a number of encoding units, each of which includes an encoding interactive convolution Conv and an expansion module Ex; the decoding module includes a number of decoding units, an inverse expansion module Dex independent of the number of decoding units, and a deprecoding convolution Dpconv, each of which includes a decoding interactive convolution Dconv and an inverse expansion module Dex; wherein:

[0159] The interactive element output by the interactive calculation module Com is fed to each encoding unit included in the encoding module to perform encoding interactive convolution calculation; at the same time, the interactive element is also fed to each decoding unit included in the decoding module to perform decoding interactive convolution calculation;

[0160] The output of each decoding unit included in the decoding module is fed to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is fed to the de-precoding convolution Dpconv.

[0161] According to the system of the second aspect, in the precoding stage, the precoding module is called to execute:

[0162] receiving a first image and a second image as multi-source data;

[0163] The multi-source data is pre-encoded using the pre-coding convolution Pconv, and the pre-coding result is input into the extension module Ex. The output of the extension module Ex is used as the pre-coding result, which is expressed as:

[0164]

[0165] Among them, C Pi represents the precoding result, P i Represents multi-source data, Represents the initialized convolution basis;

[0166] At the same time, the interactive calculation module Com calculates the interactive element B based on the first image and the second image i,j , interaction element B i,j It is then fed to the encoding module and the decoding module; the interaction element B i,j It is expressed as:

[0167]

[0168] in, Represents the transposed form, l represents the number of vector rows, p represents the number of vector columns, and n represents the number of channels.

[0169] According to the system of the second aspect, in the encoding stage, the encoding module is called to execute:

[0170] Receiving a precoding result, the precoding result is input into a coding unit, each coding unit is connected in series, the output of the previous coding unit is used as the input of the next coding unit, and the output of the last coding unit is used as the coding result of the coding module;

[0171] For each coding unit, the encoding process is expressed as:

[0172]

[0173] Among them, C C' Represents the input of the current coding unit, C C represents the encoding result of the current coding unit, ρ represents the activation function, and Φ represents the expansion basis.

[0174] According to the system of the second aspect, in the decoding stage, the decoding module is called to execute:

[0175] Receiving the encoding result, the encoding result is input to the decoding unit, each decoding unit is connected in series, and the output of the previous decoding unit is used as the input of the next decoding unit;

[0176] The output of each decoding unit is input to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is input to the de-precoding convolution Dpconv, and the output of the de-precoding convolution Dpconv and the output of the last encoding unit are input to the de-precoding convolution Dpconv;

[0177] For each decoding unit, the decoding process is expressed as:

[0178]

[0179] in, Represents the input of the current decoding unit, Indicates the decoding result of the current decoding unit;

[0180] The output of the deprecoded convolution Dpconv is expressed as:

[0181]

[0182] Among them, P i and P j Respectively represent the results of the first image and the second image being processed by the image processing system, represents the output of the last encoding unit, Represents the vectorized result after performing dual basis on the convolution basis.

[0183] The third aspect of the present invention discloses an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the image processing method for multi-source data described in the first aspect of the present disclosure are implemented.

[0184] Figure 4 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 4 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the electronic device, or an external keyboard, touch pad or mouse, etc.

[0185] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0186] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the image processing method for multi-source data described in the first aspect of the present disclosure are implemented.

[0187] In summary, the present invention designs a new convolutional neural network model for image processing from the perspective of multi-data interaction and lightweight. The model is mainly a set of encoder-decoder structures, which cleverly use convolution to complete data interaction during encoding and decoding, establish connections between input data, and make full use of the advantages of multiple data. In addition, the model has a simple structure. Under the premise of reasonable use of multiple data, only a few layers of network structure are needed to have the processing power of the classic network.

[0188] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. An image processing method for multi-source data, characterized in that: The method comprises: Step S1, acquiring a first image and a second image as multi-source data; Step S2, constructing and training an image processing system; Step S3: Process the multi-source data using the trained image processing system. in: The image processing system includes a precoding module, a coding module and a decoding module; the precoding module includes a precoding convolution Pconv, an expansion module Ex, and an interactive computing module Com; the coding module includes a plurality of coding units, each of which includes a coding interactive convolution Conv and an expansion module Ex; the decoding module includes a plurality of decoding units, an inverse expansion module Dex independent of the plurality of decoding units, and a deprecoding convolution Dpconv, each of which includes a decoding interactive convolution Dconv and an inverse expansion module Dex; The interactive element output by the interactive calculation module Com is fed to each encoding unit included in the encoding module to perform encoding interactive convolution calculation; at the same time, the interactive element is also fed to each decoding unit included in the decoding module to perform decoding interactive convolution calculation; The output of each decoding unit included in the decoding module is fed to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is fed to the de-precoding convolution Dpconv.

2. The image processing method for multi-source data according to claim 1, characterized in that: In the pre-encoding phase, the pre-encoding module is called to perform: receiving a first image and a second image as multi-source data; The multi-source data is pre-encoded using the pre-coding convolution Pconv, and the pre-coding result is input into the extension module Ex. The output of the extension module Ex is used as the pre-coding result, which is expressed as: Among them, C Pi represents the precoding result, P i Represents multi-source data, Represents the initialized convolution basis; At the same time, the interactive calculation module Com calculates the interactive element B based on the first image and the second image i,j , interaction element B i,j It is then fed to the encoding module and the decoding module; the interaction element B i,j It is expressed as: in, Represents the transposed form, l represents the number of vector rows, p represents the number of vector columns, and n represents the number of channels.

3. The image processing method for multi-source data according to claim 2, characterized in that: During the encoding phase, the encoding module is called to execute: Receiving a precoding result, the precoding result is input into a coding unit, each coding unit is connected in series, the output of the previous coding unit is used as the input of the next coding unit, and the output of the last coding unit is used as the coding result of the coding module; For each coding unit, the encoding process is expressed as: Among them, C C' Represents the input of the current coding unit, C C represents the encoding result of the current coding unit, ρ represents the activation function, and Φ represents the expansion basis.

4. The image processing method for multi-source data according to claim 3, characterized in that: During the decoding phase, the decoding module is called to perform: Receiving the encoding result, the encoding result is input to the decoding unit, each decoding unit is connected in series, and the output of the previous decoding unit is used as the input of the next decoding unit; The output of each decoding unit is input to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is input to the de-precoding convolution Dpconv, and the output of the de-precoding convolution Dpconv and the output of the last encoding unit are input to the de-precoding convolution Dpconv; For each decoding unit, the decoding process is expressed as: in, Represents the input of the current decoding unit, Indicates the decoding result of the current decoding unit; The output of the deprecoded convolution Dpconv is expressed as: Among them, P i and P j Respectively represent the results of the first image and the second image being processed by the image processing system, represents the output of the last encoding unit, Represents the vectorized result after performing dual basis on the convolution basis.

5. An image processing system for multi-source data, characterized in that: The system is used to receive a first image and a second image as multi-source data, and process the multi-source data using a trained image processing system; the system comprises: a pre-coding module, a coding module and a decoding module; wherein: The precoding module includes a precoding convolution Pconv, an expansion module Ex, and an interactive calculation module Com; the encoding module includes a number of encoding units, each of which includes an encoding interactive convolution Conv and an expansion module Ex; the decoding module includes a number of decoding units, an inverse expansion module Dex independent of the number of decoding units, and a deprecoding convolution Dpconv, each of which includes a decoding interactive convolution Dconv and an inverse expansion module Dex; wherein: The interactive element output by the interactive calculation module Com is fed to each encoding unit included in the encoding module to perform encoding interactive convolution calculation; at the same time, the interactive element is also fed to each decoding unit included in the decoding module to perform decoding interactive convolution calculation; The output of each decoding unit included in the decoding module is fed to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is fed to the de-precoding convolution Dpconv.

6. The image processing system for multi-source data according to claim 5, characterized in that: In the pre-encoding phase, the pre-encoding module is called to perform: receiving a first image and a second image as multi-source data; The multi-source data is pre-encoded using the pre-coding convolution Pconv, and the pre-coding result is input into the extension module Ex. The output of the extension module Ex is used as the pre-coding result, which is expressed as: Among them, C Pi represents the precoding result, P i Represents multi-source data, Represents the initialized convolution basis; At the same time, the interactive calculation module Com calculates the interactive element B based on the first image and the second image i,j , interaction element B i,j It is then fed to the encoding module and the decoding module; the interaction element B i,j It is expressed as: in, Represents the transposed form, l represents the number of vector rows, p represents the number of vector columns, and n represents the number of channels.

7. The image processing system for multi-source data according to claim 6, characterized in that: During the encoding phase, the encoding module is called to execute: Receiving a precoding result, the precoding result is input into a coding unit, each coding unit is connected in series, the output of the previous coding unit is used as the input of the next coding unit, and the output of the last coding unit is used as the coding result of the coding module; For each coding unit, the encoding process is expressed as: Among them, C C' Represents the input of the current coding unit, C C represents the encoding result of the current coding unit, ρ represents the activation function, and Φ represents the expansion basis.

8. The image processing system for multi-source data according to claim 7, characterized in that: During the decoding phase, the decoding module is called to perform: Receiving the encoding result, the encoding result is input to the decoding unit, each decoding unit is connected in series, and the output of the previous decoding unit is used as the input of the next decoding unit; The output of each decoding unit is input to the de-expansion module Dex independent of several decoding units in the decoding module, and the output of the de-expansion module Dex independent of several decoding units is input to the de-precoding convolution Dpconv, and the output of the de-precoding convolution Dpconv and the output of the last encoding unit are input to the de-precoding convolution Dpconv; For each decoding unit, the decoding process is expressed as: in, Represents the input of the current decoding unit, Indicates the decoding result of the current decoding unit; The output of the deprecoded convolution Dpconv is expressed as: Among them, P i and P j Respectively represent the results of the first image and the second image being processed by the image processing system, represents the output of the last encoding unit, Represents the vectorized result after performing dual basis on the convolution basis.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the image processing method for multi-source data described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the image processing method for multi-source data described in any one of claims 1 to 4 are implemented.

Images