An image processing method and system for multi-source data

Abstract

The application provides an image processing method and system for multi-source data, and belongs to the technical field of image processing. The application designs a new convolutional neural network model from the perspective of multi-data interaction and light weight, which is used for image processing, such as noise reduction. The model is mainly a group of encoder-decoder structure, which skillfully uses convolution to complete the interaction of data, establishes the connection between input data, and fully utilizes the advantages of multiple data during the encoding and decoding process. In addition, the model structure is simple, and under the premise of reasonable use of multiple data, only a few network structures can have the processing capacity of a classic network.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to an image processing method and system for multi-source data. Background Technology

[0002] Deep learning is a hot research area, and with its continuous development, it has played a vital role in many fields. Convolutional Neural Networks (CNNs), as one of the most important methods in deep learning, have achieved excellent results in many tasks due to their superior feature extraction capabilities. In processing data provided by distributed platforms, CNNs offer superior model accuracy and robustness compared to traditional methods, making them a widely used approach. However, most existing models are application studies of classic networks, using single-input networks to extract features from multiple datasets before further processing. This approach does not consider the inherent relationships and differences between multiple datasets, failing to fully leverage the advantages of multiple datasets. While blindly deepening the network model may improve performance, it increases model complexity and optimization difficulty. Therefore, designing more reasonable distributed data processing networks that balance network performance and lightweight design to improve usability is a pressing issue.

[0003] From the perspective of existing network models, the technique of using multiple parallel network modules to extract features from multiple data sets, and then connecting them to a feature processing module to process multiple data sets, is similar to the technology in this patent. It mainly consists of two parts: first, designing a set of networks to extract features from the data separately; second, inputting the output of the feature extraction modules into the subsequent feature processing module, which can be a network or a traditional optimization model. This processing method cannot effectively establish connections between data during feature extraction, fully utilize the advantages of multiple input data, and the combined application of multiple networks also increases the difficulty of training the network model. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by proposing an image processing scheme for multi-source data.

[0005] The first aspect of this invention proposes an image processing method for multi-source data, the method comprising:

[0006] Step S1: Obtain the first image and the second image as multi-source data;

[0007] Step S2: Build and train the image processing system;

[0008] Step S3: Process the multi-source data using a trained image processing system.

[0009] in:

[0010] The image processing system includes a precoding module, an encoding module, and a decoding module. The precoding module includes a precoding convolution Pconv, an extension module Ex, and an interactive computation module Com. The encoding module includes several encoding units, each of which includes an encoding interactive convolution Conv and an extension module Ex. The decoding module includes several decoding units, an inverse extension module Dex independent of the decoding units, and a deprecoding convolution Dpconv. Each decoding unit includes a decoding interactive convolution Dconv and an inverse extension module Dex.

[0011] The interactive elements output by the interactive computation module Com are fed into each encoding unit in the encoding module for encoding interactive convolution computation; at the same time, the interactive elements are also fed into each decoding unit in the decoding module for decoding interactive convolution computation.

[0012] The output of each decoding unit in the decoding module is fed into the inverse expansion module Dex, which is independent of several decoding units in the decoding module. The output of the inverse expansion module Dex, which is independent of several decoding units, is fed into the deprecoding convolution Dpconv.

[0013] According to the method in the first aspect, during the precoding stage, the precoding module is invoked to perform:

[0014] Receive the first and second images as multi-source data;

[0015] The multi-source data is pre-encoded using a precoding convolution Pconv. The pre-encoding result is input into an extension module Ex, and the output of the extension module Ex is used as the pre-encoding result, represented as follows:

[0016]

[0017] Among them, C Pi P represents the precoding result. i Indicates multi-source data, Indicates the initial convolutional basis;

[0018] Simultaneously, the interactive computing module Com calculates the interactive element B based on the first and second images. i,j Interactive element B i,j It is then fed into the encoding and decoding modules; Interactive element B i,j Represented as:

[0019]

[0020] in, This represents the transpose form, where l represents the number of rows in the vector, p represents the number of columns in the vector, and n represents the number of channels.

[0021] According to the method in the first aspect, during the encoding phase, the encoding module is invoked to execute:

[0022] The precoding result is received and input to the encoding unit. The encoding units are connected in series. The output of the previous encoding unit is used as the input of the next encoding unit, and the output of the last encoding unit is used as the encoding result of the encoding module.

[0023] For each coding unit, the coding process is represented as follows:

[0024]

[0025] Among them, C C' C represents the input of the current encoding unit. C This represents the encoding result of the current coding unit, ρ represents the activation function, and Φ represents the extension base.

[0026] According to the method in the first aspect, during the decoding phase, the decoding module is invoked to execute:

[0027] The encoding result is received and input to the decoding unit. The decoding units are connected in series, and the output of the previous decoding unit is used as the input of the next decoding unit.

[0028] The output of each decoding unit is input to the inverse expansion module Dex, which is independent of several decoding units in the decoding module. The output of the inverse expansion module Dex, which is independent of several decoding units, is input to the deprecoding convolution Dpconv. The output of the deprecoding convolution Dpconv and the output of the last encoding unit are input to the deprecoding convolution Dpconv.

[0029] For each decoding unit, the decoding process is represented as follows:

[0030]

[0031] in, This represents the input of the current decoding unit. This indicates the decoding result of the current decoding unit;

[0032] The output of the de-encoding convolution Dpconv is represented as:

[0033]

[0034] Among them, P i and P j These represent the results of the first and second images being processed by the image processing system, respectively. This indicates the output of the last encoded unit. This represents the vectorization result after applying dual basis to the convolutional basis.

[0035] A second aspect of the present invention provides an image processing system for multi-source data. The system receives a first image and a second image as multi-source data and processes the multi-source data using a trained image processing system. The system includes a precoding module, an encoding module, and a decoding module; wherein:

[0036] The precoding module includes a precoding convolution Pconv, an extension module Ex, and an interactive computation module Com; the encoding module includes several encoding units, each including an encoding interactive convolution Conv and an extension module Ex; the decoding module includes several decoding units, an inverse extension module Dex independent of the decoding units, and a deprecoding convolution Dpconv, each decoding unit including a decoding interactive convolution Dconv and an inverse extension module Dex; wherein:

[0037] The interactive elements output by the interactive computation module Com are fed into each encoding unit in the encoding module for encoding interactive convolution computation; at the same time, the interactive elements are also fed into each decoding unit in the decoding module for decoding interactive convolution computation.

[0038] The output of each decoding unit in the decoding module is fed into the inverse expansion module Dex, which is independent of several decoding units in the decoding module. The output of the inverse expansion module Dex, which is independent of several decoding units, is fed into the deprecoding convolution Dpconv.

[0039] According to the system in the second aspect, during the precoding stage, the precoding module is invoked to execute:

[0040] Receive the first and second images as multi-source data;

[0041] The multi-source data is pre-encoded using a precoding convolution Pconv. The pre-encoding result is input into an extension module Ex, and the output of the extension module Ex is used as the pre-encoding result, represented as follows:

[0042]

[0043] Among them, C Pi P represents the precoding result. i Indicates multi-source data, Indicates the initial convolutional basis;

[0044] Simultaneously, the interactive computing module Com calculates the interactive element B based on the first and second images. i,j Interactive element B i,j It is then fed into the encoding and decoding modules; Interactive element B i,j Represented as:

[0045]

[0046] in, This represents the transpose form, where l represents the number of rows in the vector, p represents the number of columns in the vector, and n represents the number of channels.

[0047] According to the system in the second aspect, during the encoding phase, the encoding module is invoked to execute:

[0048] The precoding result is received and input to the encoding unit. The encoding units are connected in series. The output of the previous encoding unit is used as the input of the next encoding unit, and the output of the last encoding unit is used as the encoding result of the encoding module.

[0049] For each coding unit, the coding process is represented as follows:

[0050]

[0051] Among them, C C' C represents the input of the current encoding unit. C This represents the encoding result of the current coding unit, ρ represents the activation function, and Φ represents the extension base.

[0052] According to the system in the second aspect, during the decoding phase, the decoding module is invoked to execute:

[0053] The encoding result is received and input to the decoding unit. The decoding units are connected in series, and the output of the previous decoding unit is used as the input of the next decoding unit.

[0054] The output of each decoding unit is input to the inverse expansion module Dex, which is independent of several decoding units in the decoding module. The output of the inverse expansion module Dex, which is independent of several decoding units, is input to the deprecoding convolution Dpconv. The output of the deprecoding convolution Dpconv and the output of the last encoding unit are input to the deprecoding convolution Dpconv.

[0055] For each decoding unit, the decoding process is represented as follows:

[0056]

[0057] in, This represents the input of the current decoding unit. This indicates the decoding result of the current decoding unit;

[0058] The output of the de-encoding convolution Dpconv is represented as:

[0059]

[0060] Among them, P i and P jThese represent the results of the first and second images being processed by the image processing system, respectively. This indicates the output of the last encoded unit. This represents the vectorization result after applying dual basis to the convolutional basis.

[0061] A third aspect of this invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the image processing method for multi-source data described in the first aspect of this disclosure.

[0062] A fourth aspect of this invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image processing method for multi-source data described in the first aspect of this disclosure.

[0063] In summary, this invention designs a novel convolutional neural network model for image processing from the perspectives of multi-data interaction and lightweight design. This model primarily consists of an encoder-decoder structure, cleverly utilizing convolution to facilitate data interaction during encoding and decoding, establishing connections between input data, and fully leveraging the advantages of multiple data sets. Furthermore, the model has a simple structure; by rationally utilizing multiple data sets, it requires only a few network layers to achieve the processing capabilities of classic networks. Attached Figure Description

[0064] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0065] Figure 1 This is a schematic diagram of the structure of a data processing system according to an embodiment of the present invention.

[0066] Figure 2 This is a schematic diagram of image noise reduction processing according to an embodiment of the present invention.

[0067] Figure 3 This is a schematic diagram of image stitching processing according to an embodiment of the present invention.

[0068] Figure 4 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] The first aspect of this invention proposes an image processing method for multi-source data, the method comprising:

[0071] Step S1: Obtain the first image and the second image as multi-source data;

[0072] Step S2: Build and train the image processing system;

[0073] Step S3: Process the multi-source data using a trained image processing system.

[0074] in:

[0075] Image processing systems (such as) Figure 1 (As shown) includes a precoding module, an encoding module, and a decoding module; the precoding module includes a precoding convolution Pconv, an extension module Ex, and an interactive computation module Com; the encoding module includes several encoding units, each encoding unit including an encoding interactive convolution Conv and an extension module Ex; the decoding module includes several decoding units, an inverse extension module Dex independent of the several decoding units, and a deprecoding convolution Dpconv, each decoding unit including a decoding interactive convolution Dconv and an inverse extension module Dex;

[0076] The interactive elements output by the interactive computation module Com are fed into each encoding unit in the encoding module for encoding interactive convolution computation; at the same time, the interactive elements are also fed into each decoding unit in the decoding module for decoding interactive convolution computation.

[0077] The output of each decoding unit in the decoding module is fed into the inverse expansion module Dex, which is independent of several decoding units in the decoding module. The output of the inverse expansion module Dex, which is independent of several decoding units, is fed into the deprecoding convolution Dpconv.

[0078] According to the method in the first aspect, during the precoding stage, the precoding module is invoked to perform:

[0079] Receive the first and second images as multi-source data;

[0080] The multi-source data is pre-encoded using a precoding convolution Pconv. The pre-encoding result is input into an extension module Ex, and the output of the extension module Ex is used as the pre-encoding result, represented as follows:

[0081]

[0082] Among them, C Pi P represents the precoding result. i Indicates multi-source data, Indicates the initial convolutional basis;

[0083] Simultaneously, the interactive computing module Com calculates the interactive element B based on the first and second images. i,j Interactive element B i,j It is then fed into the encoding and decoding modules; Interactive element B i,j Represented as:

[0084]

[0085] in, This represents the transpose form, where l represents the number of rows in the vector, p represents the number of columns in the vector, and n represents the number of channels.

[0086] According to the method in the first aspect, during the encoding phase, the encoding module is invoked to execute:

[0087] The precoding result is received and input to the encoding unit. The encoding units are connected in series. The output of the previous encoding unit is used as the input of the next encoding unit, and the output of the last encoding unit is used as the encoding result of the encoding module.

[0088] For each coding unit, the coding process is represented as follows:

[0089]

[0090] Among them, C C' C represents the input of the current encoding unit. C This represents the encoding result of the current coding unit, ρ represents the activation function, and Φ represents the extension base.

[0091] According to the method in the first aspect, during the decoding phase, the decoding module is invoked to execute:

[0092] The encoding result is received and input to the decoding unit. The decoding units are connected in series, and the output of the previous decoding unit is used as the input of the next decoding unit.

[0093] The output of each decoding unit is input to the inverse expansion module Dex, which is independent of several decoding units in the decoding module. The output of the inverse expansion module Dex, which is independent of several decoding units, is input to the deprecoding convolution Dpconv. The output of the deprecoding convolution Dpconv and the output of the last encoding unit are input to the deprecoding convolution Dpconv.

[0094] For each decoding unit, the decoding process is represented as follows:

[0095]

[0096] in, This represents the input of the current decoding unit. This indicates the decoding result of the current decoding unit;

[0097] The output of the de-encoding convolution Dpconv is represented as:

[0098]

[0099] Among them, P i and P j These represent the results of the first and second images being processed by the image processing system, respectively. This indicates the output of the last encoded unit. This represents the vectorization result after applying dual basis to the convolutional basis.

[0100] Example 1

[0101] Taking two inputs as an example, such as Figure 1 As shown, based on Hankel convolution, interactive convolution was innovatively proposed, and a network was constructed based on this, which can be divided into three parts: pre-encoding: first, the input data is pre-encoded, which is the basis for the interactive convolution in the subsequent encoder; encoder: through interactive convolution, the interaction between data is completed during the encoding process; decoder: still in the form of interactive convolution, the decoding process is realized by selecting the dual basis in the encoder.

[0102] The model uses Hankel convolution for input P∈R. nl×p and convolution basis Ψ∈R p×q Each column of the input can be viewed as l n-dimensional vectors:

[0103]

[0104] Hankel convolution can be represented as:

[0105]

[0106] in, This indicates that the matrix is ​​flipped column by column. for:

[0107]

[0108] Pre-convolution is performed on the input to compute the interaction elements. The convolution basis Ψ is initialized, and a specific input is selected for Hankel convolution, specifically in the form of:

[0109]

[0110] Then, the interaction element B is calculated by pairwise input pairing. i,j The calculation method is as follows:

[0111]

[0112] in, The special transpose of the input can be represented as:

[0113]

[0114] To ensure the uniqueness of the inverse calculation, the input data needs to be vectorized before input to ensure that the input dimension satisfies l≥np. Then, the interaction elements and the pre-convolution result are input into the encoder for the next step of computation.

[0115] The output obtained from the pre-convolution is subjected to interactive convolution, and convolutional basis expansion is performed to extract deeper features. Interactive convolution is a Hankel convolution performed between the corresponding interactive element and the output of the previous convolution, which can be represented as... Through interactive convolutions, the model can switch the active input data for a given layer. The number of interactive convolutions and the participating interactive elements can be flexibly designed according to the amount of input and design requirements. After each interactive convolution, it is directly connected to the expansion element Φ. i After performing the calculation, select an appropriate activation function to perform the operation.

[0116] Taking input 1 and input 2 as examples, the output of the pre-convolution is C1, and the activation function is represented by ρ(·). A complete interactive convolution and convolution expansion can be represented as:

[0117]

[0118] The encoder output is decoded. Corresponding to the encoder, decoding also includes two parts: deconvolution and the inverse process of spreading.

[0119] First, find the extended basis. and convolutional basis Meet the conditions After the inverse transform of the extended module, interactive convolution is performed, and the convolution result is processed using an appropriate activation function. The output of the previous layer is... Taking input 1 and input 2 as examples again, the input that takes effect in the previous layer is P1, and the above process can be represented as:

[0120]

[0121] After several interactive convolutions and inverse transformations of the expansion module, the decoded output is finally obtained through deconvolution, with the same dimension as the input.

[0122] And the dual basis of the convolutional basis is locally vectorized to obtain Specifically, it is expressed as follows:

[0123]

[0124] The entire deconvolution is represented as:

[0125]

[0126] Other inputs can be obtained through interactive convolutions:

[0127]

[0128] To make it easier to understand, let's take two inputs as an example. An interaction occurs once during the encoding and once during the decoding process. The entire process can be represented as follows:

[0129]

[0130] Example 2

[0131] To verify the superiority of the proposed network model, a denoising experiment was conducted using two preprocessed similar images (the purpose of preprocessing is to improve the low-rank property of the input data and optimize the model's processing capability; here, only data concatenation is used as an example). Figure 2 As shown.

[0132] Preprocessing involves dividing several identical input images, each with the same level of noise added, into two groups. Each group is then cut into 300×5 image blocks. After stitching, two low-rank outputs are obtained, such as... Figure 3 As shown.

[0133] The preprocessed training set images are input into the model for training. The target is clean images without added noise. The loss function chosen is the L2 loss, specifically expressed as:

[0134]

[0135] Where m and n are the dimensions of the image, and (i,j) is the corresponding position. The trained model is obtained through training.

[0136] The images in the test set with added noise are preprocessed and then input into the trained model to obtain denoised image patches. To avoid performance advantages due to noise statistics, one of the denoised images is selected and stitched together to the original image size to obtain the denoised image.

[0137] The programming was completed using PyTorch in PyCharm, on a portable laptop equipped with a 4060 graphics card. For training, ten 2040×1360 images from the DIV2K dataset were selected, and simulated noise of the same level was added to generate the training set. For the test set, two grayscale image sets, Set12 and BSD68, were selected, also with simulated noise added as input. To meet the input size requirements, the image size for Set12 was modified to 512×512; areas that were not large enough were padded with duplicates. The network training parameters are shown in Table 1.

[0138] Table 1: Main Technical Specifications Used

[0139]

[0140] To highlight the performance of the proposed network, DnCNN, BM3D, Restormer, and BRDNet models were selected for comparison. Due to the limited training set, Restormer performed poorly, so the original author's trained model was used as the comparison result.

[0141] Table 2: Results of the Comprehensive Noise Reduction Experiment

[0142]

[0143] The experimental results are shown in Table 2. The comparison reveals that the proposed model significantly improves upon existing methods, fully leveraging the advantages of multiple datasets and effectively enhancing processing performance. Furthermore, the model requires less time to process images; processing a 512×512 image takes less than 0.18 seconds, demonstrating excellent processing efficiency.

[0144] Example 3

[0145] To further demonstrate the generalization ability of the proposed model, we will again use two inputs as an example and conduct experiments using two preprocessed images that are clearly different from each other. The process is as follows:

[0146] Preprocessing involves dividing the same input images, which have been subjected to different levels of noise, into two groups. Each group is then cut into 300×5 image blocks and stitched together to obtain two low-rank outputs.

[0147] The preprocessed training set images are input into the model for training. The goal is to obtain clean images without added noise. The L2 loss is used as the loss function to obtain the trained model.

[0148] The images added to the test set are preprocessed and then input into the trained model to obtain denoised image patches. To avoid performance advantages due to noise statistics, one of the denoised images is selected and stitched together to the original image size to obtain the denoised image.

[0149] The experimental equipment was the same as in Example 1. For training, ten 2040×1360 images from the DIV2K dataset were selected, and two different levels of simulated noise were added to generate the training set. For the test set, two grayscale image sets, Set12 and BSD68, were selected, also with simulated noise added as input. To meet the input size requirements, the image size for Set12 was modified to 512×512; areas that were not large enough were padded with duplicate parts. The network training parameters are shown in Table 3.

[0150] Table 3: Main Technical Specifications Used

[0151]

[0152] For the denoising experiments with different input noise levels, the noise levels for training and testing were set to σ = 15 and σ = 25, respectively, and the results are shown in Table 4.

[0153] Table 4: Experimental Results of Complementary Noise Reduction

[0154]

[0155] Experiments show that the model can still maintain high processing performance when inputting the same image with different levels of background noise. Although the result is slightly lower than that of the same level of noise in Example 1, it still shows strong generalization ability.

[0156] As can be seen, this invention provides an interactive bridge for multiple input data, making fuller use of the advantages of multiple data; it extracts features from multiple data using a set of parameters, reducing model complexity and improving image processing efficiency; by using reasonable low-rank preprocessing methods, the model's processing effect is higher than that of some existing models; model training is more intuitive, eliminating the need to learn specific noise to obtain denoised images, and image reconstruction can be completed directly; the model has better generalization ability, and can effectively utilize complementary features to maintain good accuracy when processing differential data.

[0157] A second aspect of the present invention provides an image processing system for multi-source data. The system receives a first image and a second image as multi-source data and processes the multi-source data using a trained image processing system. The system includes a precoding module, an encoding module, and a decoding module; wherein:

[0158] The precoding module includes a precoding convolution Pconv, an extension module Ex, and an interactive computation module Com; the encoding module includes several encoding units, each including an encoding interactive convolution Conv and an extension module Ex; the decoding module includes several decoding units, an inverse extension module Dex independent of the decoding units, and a deprecoding convolution Dpconv, each decoding unit including a decoding interactive convolution Dconv and an inverse extension module Dex; wherein:

[0159] The interactive elements output by the interactive computation module Com are fed into each encoding unit in the encoding module for encoding interactive convolution computation; at the same time, the interactive elements are also fed into each decoding unit in the decoding module for decoding interactive convolution computation.

[0160] The output of each decoding unit in the decoding module is fed into the inverse expansion module Dex, which is independent of several decoding units in the decoding module. The output of the inverse expansion module Dex, which is independent of several decoding units, is fed into the deprecoding convolution Dpconv.

[0161] According to the system in the second aspect, during the precoding stage, the precoding module is invoked to execute:

[0162] Receive the first and second images as multi-source data;

[0163] The multi-source data is pre-encoded using a precoding convolution Pconv. The pre-encoding result is input into an extension module Ex, and the output of the extension module Ex is used as the pre-encoding result, represented as follows:

[0164]

[0165] Among them, C Pi P represents the precoding result. i Indicates multi-source data, Indicates the initial convolutional basis;

[0166] Simultaneously, the interactive computing module Com calculates the interactive element B based on the first and second images. i,j Interactive element B i,j It is then fed into the encoding and decoding modules; Interactive element B i,j Represented as:

[0167]

[0168] in, This represents the transpose form, where l represents the number of rows in the vector, p represents the number of columns in the vector, and n represents the number of channels.

[0169] According to the system in the second aspect, during the encoding phase, the encoding module is invoked to execute:

[0170] The precoding result is received and input to the encoding unit. The encoding units are connected in series. The output of the previous encoding unit is used as the input of the next encoding unit, and the output of the last encoding unit is used as the encoding result of the encoding module.

[0171] For each coding unit, the coding process is represented as follows:

[0172]

[0173] Among them, C C' C represents the input of the current encoding unit. C This represents the encoding result of the current coding unit, ρ represents the activation function, and Φ represents the extension base.

[0174] According to the system in the second aspect, during the decoding phase, the decoding module is invoked to execute:

[0175] The encoding result is received and input to the decoding unit. The decoding units are connected in series, and the output of the previous decoding unit is used as the input of the next decoding unit.

[0176] The output of each decoding unit is input to the inverse expansion module Dex, which is independent of several decoding units in the decoding module. The output of the inverse expansion module Dex, which is independent of several decoding units, is input to the deprecoding convolution Dpconv. The output of the deprecoding convolution Dpconv and the output of the last encoding unit are input to the deprecoding convolution Dpconv.

[0177] For each decoding unit, the decoding process is represented as follows:

[0178]

[0179] in, This represents the input of the current decoding unit. This indicates the decoding result of the current decoding unit;

[0180] The output of the de-encoding convolution Dpconv is represented as:

[0181]

[0182] Among them, P i and P j These represent the results of the first and second images being processed by the image processing system, respectively. This indicates the output of the last encoded unit. This represents the vectorization result after applying dual basis to the convolutional basis.

[0183] A third aspect of this invention discloses an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the image processing method for multi-source data described in the first aspect of this disclosure.

[0184] Figure 4 This is a structural diagram of an electronic device according to an embodiment of the present invention, such as... Figure 4 As shown, the electronic device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, Near Field Communication (NFC), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0185] Those skilled in the art will understand that Figure 4 The structure shown is merely a structural diagram of the part related to the technical solution of this disclosure and does not constitute a limitation on the electronic device to which the solution of this application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0186] A fourth aspect of this invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image processing method for multi-source data described in the first aspect of this disclosure.

[0187] In summary, this invention designs a novel convolutional neural network model for image processing from the perspectives of multi-data interaction and lightweight design. This model primarily consists of an encoder-decoder structure, cleverly utilizing convolution to facilitate data interaction during encoding and decoding, establishing connections between input data, and fully leveraging the advantages of multiple data sets. Furthermore, the model has a simple structure; by rationally utilizing multiple data sets, it requires only a few network layers to achieve the processing capabilities of classic networks.

[0188] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for image processing of multi-source data, characterized in that, The method comprises: Step S1, acquiring a first image and a second image as multi-source data; Step S2, constructing and training an image processing system; Step S3, processing the multi-source data by using the trained image processing system; Wherein: The image processing system comprises a pre-encoding module, an encoding module and a decoding module; the pre-encoding module comprises a pre-encoding convolution Pconv, an expansion module Ex and an interaction calculation module Com; the encoding module comprises a plurality of encoding units, each of which comprises an encoding interaction convolution Conv and an expansion module Ex; the decoding module comprises a plurality of decoding units, an inverse expansion module Dex independent of the plurality of decoding units and a de-pre-encoding convolution Dpconv, each of which comprises a decoding interaction convolution Dconv and an inverse expansion module Dex; The interaction element output by the interaction calculation module Com is fed to each encoding unit contained in the encoding module for encoding interaction convolution calculation; at the same time, the interaction element is also fed to each decoding unit contained in the decoding module for decoding interaction convolution calculation; The output of each decoding unit contained in the decoding module is fed to the inverse expansion module Dex independent of the plurality of decoding units in the decoding module, and the output of the inverse expansion module Dex independent of the plurality of decoding units is fed to the de-pre-encoding convolution Dpconv; In the pre-encoding stage, the pre-encoding module is called to perform: Receiving the first image and the second image as multi-source data; Pre-encoding the multi-source data by using the pre-encoding convolution Pconv, inputting the pre-encoding result to the expansion module Ex, and taking the output of the expansion module Ex as the pre-encoding result, which is expressed as: wherein C Pi represents a precoding result, P i represents multi-source data, represents an initialized convolutional base; The simultaneous interaction computing module Com computes an interaction element B based on the first image and the second image i,j , the interaction element B i,j is then fed to the encoding module and the decoding module; the interaction element B i,j is represented as: wherein denotes the transpose, and n denotes the number of channels.

2. The image processing method for multi-source data according to claim 1, characterized in that, In the encoding stage, the encoding module is called to perform: Receiving the pre-encoding result, which is input to the encoding unit, each encoding unit is connected in series, the output of the previous encoding unit is taken as the input of the next encoding unit, and the output of the last encoding unit is taken as the encoding result of the encoding module; For each encoding unit, the encoding process is expressed as: where C C′ denotes the input of the current coding unit, C C denotes the encoding result of the current coding unit, p denotes an activation function, and Φ denotes an expansion base.

3. The image processing method for multi-source data according to claim 2, wherein, In the decoding stage, the decoding module is called to perform: Receiving the encoding result, which is input to the decoding unit, each decoding unit is connected in series, and the output of the previous decoding unit is taken as the input of the next decoding unit; The output of each decoding unit is input to the inverse expansion module Dex independent of the plurality of decoding units in the decoding module, and the output of the inverse expansion module Dex independent of the plurality of decoding units is input to the de-pre-encoding convolution Dpconv, and the output of the de-pre-encoding convolution Dpconv and the output of the last encoding unit are input to the de-pre-encoding convolution Dpconv; For each decoding unit, the decoding process is expressed as: wherein, represents an input of the current decoding unit, represents a decoding result of the current decoding unit; The output of the de-pre-encoding convolution Dpconv is expressed as: where P i and P j represent the results of the first and second images after processing by the image processing system, respectively, represents the output of the last encoding unit, represents the vectorization result of the dual basis of the convolution basis.

4. An image processing system for multi-source data, characterized by, The system is used for receiving a first image and a second image as multi-source data, and processing the multi-source data by using the trained image processing system; the system comprises a pre-encoding module, an encoding module and a decoding module; wherein: The pre-encoding module comprises a pre-encoding convolution Pconv, an expansion module Ex, and an interaction calculation module Com; the encoding module comprises a plurality of encoding units, each of which comprises an encoding interaction convolution Conv and an expansion module Ex; the decoding module comprises a plurality of decoding units, an inverse expansion module Dex independent of the plurality of decoding units, and a de-pre-encoding convolution Dpconv, each of which comprises a decoding interaction convolution Dconv and an inverse expansion module Dex; wherein: The interaction element output by the interaction calculation module Com is fed to each encoding unit contained in the encoding module for encoding interaction convolution calculation; at the same time, the interaction element is also fed to each decoding unit contained in the decoding module for decoding interaction convolution calculation; The output of each decoding unit contained in the decoding module is fed to the inverse expansion module Dex independent of the plurality of decoding units in the decoding module, and the output of the inverse expansion module Dex independent of the plurality of decoding units is fed to the de-pre-encoding convolution Dpconv; In the pre-encoding stage, the pre-encoding module is called to perform: receiving a first image and a second image as multi-source data; pre-encoding the multi-source data by using the pre-encoding convolution Pconv, inputting the pre-encoding result to the expansion module Ex, and taking the output of the expansion module Ex as the pre-encoding result, which is expressed as: wherein C Pi represents a precoding result, P i represents multi-source data, represents an initialized convolutional base; The simultaneous interaction computing module Com computes an interaction element B based on the first image and the second image i,j , the interaction element B i,j is then fed to the encoding module and the decoding module; the interaction element B i,j is represented as: wherein denotes the transpose, and n denotes the number of channels.

5. The image processing system for multi-source data according to claim 4, wherein, In the encoding stage, the encoding module is called to perform: receiving the pre-encoding result, inputting the pre-encoding result to the encoding units, connecting the encoding units in series, taking the output of a previous encoding unit as the input of a subsequent encoding unit, and taking the output of the last encoding unit as the encoding result of the encoding module; For each encoding unit, the encoding process is expressed as: where C C′ denotes the input of the current coding unit, C C denotes the encoding result of the current coding unit, p denotes an activation function, and Φ denotes an expansion base.

6. The image processing system for multi-source data according to claim 5, wherein, In the decoding stage, the decoding module is called to perform: receiving the encoding result, inputting the encoding result to the decoding units, and connecting the decoding units in series; The output of each decoding unit is input to the inverse expansion module Dex independent of the plurality of decoding units in the decoding module, and the output of the inverse expansion module Dex independent of the plurality of decoding units is input to the de-pre-encoding convolution Dpconv, and the output of the de-pre-encoding convolution Dpconv and the output of the last encoding unit are input to the de-pre-encoding convolution Dpconv; For each decoding unit, the decoding process is expressed as: wherein, represents an input of the current decoding unit, represents a decoding result of the current decoding unit; The output of the de-pre-encoding convolution Dpconv is expressed as: wherein P i and P j represent the results of the first and second images after processing by the image processing system, respectively, represents the output of the last encoding unit, represents the vectorization result of the dual basis of the convolution basis.

7. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the image processing method for multi-source data according to any one of claims 1-3 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the image processing method for multi-source data according to any one of claims 1-3.

Images