Image processing method and device and electronic equipment

Abstract

The embodiment of the invention provides an image processing method and device and electronic equipment, and the method comprises the steps: obtaining a to-be-processed image which comprises a target object needing to be removed; inputting the first feature map corresponding to the to-be-processed image into a lightweight network converter of the image processing model for feature extraction to obtain a second feature map; inputting the second feature map into a multi-scale feature fusion network of the image processing model for feature fusion, and obtaining a first image after image reconstruction; processing the to-be-processed image by using the first image to obtain a target image in which the target object is filtered; wherein the lightweight network converter comprises at least two network conversion modules which are connected in series and are used for extracting shallow layer features of the to-be-processed image. According to the embodiment of the invention, a clearer target image can be generated, and the image processing effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device and electronic equipment. Background Art

[0002] In image processing processes such as image recognition and image analysis, if there are obstructions such as rain streaks, watermarks, and noise in the image to be processed, the image quality will be reduced, causing the recognition of the target object in the image to decrease, affecting the processing results.

[0003] If the goal is to remove occlusions in a video, since the video contains continuous image frame information, the occlusions can be removed based on the timing information of the continuous image frames. However, since a single image lacks timing information, how to remove occlusions in a single image has become a problem that needs to be solved in related industries.

[0004] In the related art, the method of removing image occlusions has the problem of blurred local details in the generated image, and the texture features of the image are difficult to retain, resulting in poor image processing effect. Summary of the invention

[0005] The embodiments of the present invention provide an image processing method, device and electronic device, which can solve the problem in the related art that the processing result of removing occluders in a single image is difficult to retain texture features and the image processing effect is poor.

[0006] In order to solve the above problem, an embodiment of the present invention discloses an image processing method, which includes:

[0007] Acquire an image to be processed, wherein the image to be processed contains a target object that needs to be removed in a preset image processing model;

[0008] Inputting the first feature map corresponding to the image to be processed into the lightweight network converter of the image processing model for feature extraction to obtain a second feature map;

[0009] Inputting the second feature map into the multi-scale feature fusion network of the image processing model for feature fusion, and obtaining a first image after image reconstruction;

[0010] Using the first image to process the image to be processed, to obtain a target image from which the target object is filtered out;

[0011] The lightweight network converter includes at least two network conversion modules connected in series for refining shallow features of the image to be processed.

[0012] On the other hand, an embodiment of the present invention discloses an image processing device, the device comprising:

[0013] An image acquisition module is used to acquire an image to be processed, wherein the image to be processed contains a target object that needs to be removed in a preset image processing model;

[0014] A first extraction module, used for inputting a first feature map corresponding to the image to be processed into a lightweight network converter of the image processing model for feature extraction to obtain a second feature map;

[0015] A fusion and reconstruction module, used for inputting the second feature map into the multi-scale feature fusion network of the image processing model for feature fusion, and obtaining a first image after image reconstruction;

[0016] An image processing module, used to process the image to be processed using the first image to obtain a target image from which the target object is filtered out;

[0017] The lightweight network converter includes at least two network conversion modules connected in series for refining shallow features of the image to be processed.

[0018] On the other hand, an embodiment of the present invention further discloses an electronic device, which includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to execute the aforementioned image processing method by one or more processors.

[0019] The embodiment of the present invention further discloses a readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the aforementioned image processing method.

[0020] The embodiments of the present invention include the following advantages:

[0021] An embodiment of the present invention provides an image processing method, which converts an image to be processed into a feature space by utilizing a feature extraction network in an image processing model, thereby replacing a feature extraction method based on prior information in related technologies. In addition, the embodiment of the present invention extracts global image features corresponding to a single image through a network conversion module connected in series in a lightweight network converter, and fuses the position details and spatial context information of the image through a multi-scale feature fusion network, thereby avoiding feature loss of target objects such as rain streaks, watermarks, and noise points to be filtered out in a single image, and can generate a clearer target image, thereby improving the image processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments of the present invention are briefly introduced below.

[0023] Figure 1 is a flowchart of steps of an image processing method embodiment of the present invention;

[0024] Figure 2 It is a structural schematic diagram of an image processing model of the present invention;

[0025] Figure 3 It is a structural schematic diagram of a multi-scale feature fusion network of the present invention;

[0026] Figure 4 It is a structural schematic diagram of an image reconstruction network of the present invention;

[0027] Figure 5 It is a structural schematic diagram of a network conversion module of the present invention;

[0028] Figure 6 It is a structural schematic diagram of a lightweight network converter of the present invention;

[0029] Figure 7 It is a structural schematic diagram of a residual network of the present invention;

[0030] Figure 8 is a structural block diagram of an embodiment of an image processing device of the present invention;

[0031] Fig. 9 It is a structural block diagram of an electronic device provided by an example of the present invention. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0033] Method Embodiment

[0034] The image processing method provided by the embodiment of the present invention can remove obstructions such as rain streaks and watermarks in an image.

[0035] Taking single image deraining as an example, in the relevant technology, single image deraining algorithms can be divided into model-based deraining methods and data-based deraining methods. The model-based methods mainly include those based on sparse coding and Gaussian mixture models, and the data-driven methods include those based on deep neural networks, adversarial networks, and semi-supervised networks or unsupervised networks.

[0036] Traditional model-based methods are often based on image decomposition and dictionary learning. The image is decomposed into low-frequency and high-frequency parts through bilateral filtering, and then the rain-free dictionary of the high-frequency part is obtained by using sparse representation and dictionary learning. The processed high-frequency part is then superimposed with the original low-frequency part to obtain the final rain-free image. Since this method only processes the high-frequency part of the image, there will be a small amount of rain information in the actual low-frequency part, so it is easy for rain to remain, causing the generated image to be blurred. Improved methods include the rain removal method based on the mean filter, but this method is limited by the shape of raindrops and can only detect vertically distributed raindrops with an elliptical appearance, while the shape of raindrops in actual scenes is difficult to determine. The above method can only process the rain image into a linearly separable model by establishing a mathematical model for the rain image. For complex rain images, it is easy to have problems such as rain residue and background image blur, and the algorithm has high time complexity.

[0037] Data-driven methods are often based on deep neural networks. Common application models include DerainNet, PReNet, LPNet, DCSFN, SPDNet, etc. Based on these models, although the deraining performance has been significantly improved, the main idea is to increase the depth of the convolutional neural network or use a multi-branch structure to increase network performance. This processing method will lead to a large model and difficult network training. In addition, the multi-scale information is not fully utilized in the feature extraction process. The generated images have blurred local details and it is difficult to fully retain texture features.

[0038] The embodiment of the present invention provides an image processing method, which can improve the accuracy of the processing result of removing occlusions from a single image, generate a clearer image, and improve the image processing effect.

[0039] Reference Figure 1 , shows a flowchart of an image processing method embodiment of the present invention, the method may specifically include the following steps:

[0040] Step 101: Acquire an image to be processed; the image to be processed contains a target object that needs to be removed in a preset image processing model;

[0041] Step 102: Input the first feature map corresponding to the image to be processed into the lightweight network converter of the image processing model for feature extraction to obtain a second feature map;

[0042] Step 103: input the second feature map into the multi-scale feature fusion network of the image processing model to perform feature fusion, and obtain the first image after image reconstruction;

[0043] Step 104: Use the first image to process the image to be processed to obtain a target image with the target object filtered out.

[0044] The lightweight network converter includes at least two network conversion modules connected in series for refining shallow features of the image to be processed.

[0045] The feature extraction network is used to perform preliminary feature extraction on the input image to be processed, and the extracted feature information is the feature information of the target object to be removed. For example, in the image rain removal scene, the feature extraction network is used to extract the rain streak feature information of the image to be processed. Exemplarily, the feature extraction network in the present invention can be a convolutional neural network with a convolution kernel size of 3.

[0046] The lightweight network converter is used to collect more fine-grained feature information of the target object to be removed, and further refine the global features of the image. The lightweight network converter in the present invention may include at least two lightweight network conversion modules (Light-weight-network Transformer, LWNT) connected in series, referred to as network conversion module. Exemplarily, the lightweight network converter may include 4 network conversion modules connected in series, that is, 4 LWNT structures connected in series. After the image to be processed is subjected to shallow feature extraction through the feature extraction network, a first feature map is obtained, and the first feature map is input into the lightweight network converter. The shallow features in the first feature map are refined by the 4 LWNT structures connected in series, and the global features are further refined to obtain a second feature map containing shallow position features and deep semantic features corresponding to the image to be processed.

[0047] The multi-scale feature fusion network is used to fuse the shallow position features and deep semantic features of the second feature map output by the lightweight network converter to obtain deep features with stronger expressive power. Exemplarily, the multi-scale feature fusion network in the present invention can adopt a convolutional neural network, or can adopt multiple multi-scale attention residual group modules (Multi-scale Attention Residual GroupBock, MARGB) connected in series, or, based on the residual network, a top-down feature fusion network with a horizontal connection structure is constructed as the multi-scale feature fusion network in the present invention to obtain a third feature map containing the deep features corresponding to the image to be processed.

[0048] The image reconstruction network is used to reconstruct the third feature map and output a first image containing the target object to be removed. The image reconstruction network in the present invention can use a convolutional neural network with a convolution kernel size of 3, and can also use a coordinate attention mechanism to further refine the extracted features. By constructing a position attention mechanism, not only can the cross-channel information in the third feature map be captured, but also the position and direction information can be perceived, so that the model can more accurately locate and identify the target area, refine the feature extraction process, and improve the accuracy of removing rain streaks.

[0049] In an embodiment of the present invention, an image processing model composed of an image processing feature extraction network, a lightweight network converter, a multi-scale feature fusion network, and an image reconstruction network can be trained in advance, and then the trained image processing model is used to process the image to be processed, and the target object in the image to be processed is filtered out to obtain a target image. The embodiment of the present invention converts the image to be processed into a feature space through the feature extraction network in the image processing model, replacing the feature extraction method based on prior information in the related art. In addition, the embodiment of the present invention extracts the global image features corresponding to a single image through the network conversion module connected in series in the lightweight network converter, and fuses the position details and spatial context information of the image through the multi-scale feature fusion network, thereby avoiding the loss of features of target objects such as rain streaks, watermarks, noise points, etc. to be filtered out in a single image, and can generate a clearer target image, thereby improving the image processing effect.

[0050] In an optional embodiment of the present invention, the image processing model further includes a feature extraction network, and the feature extraction network, the lightweight network converter and the multi-scale feature fusion network are sequentially connected in series; the method further includes:

[0051] Step S11, inputting the image to be processed into the feature extraction network for feature extraction to obtain the first feature map; the first feature map is used to reflect the shallow position features of the image to be processed.

[0052] The step of inputting the second feature map into the multi-scale feature fusion network of the image processing model for feature fusion, and obtaining the first image after image reconstruction, comprises:

[0053] Step S12: inputting the second feature map into the multi-scale feature fusion network for feature fusion to obtain a third feature map;

[0054] Step S13: input the third feature map into the image reconstruction network to perform image reconstruction to obtain a first image.

[0055] The step of processing the image to be processed by using the first image to obtain a target image from which the target object is filtered out includes:

[0056] Step S14: performing a subtraction operation on the image to be processed and the first image to obtain the target image.

[0057] Reference Figure 2 , which shows a schematic diagram of the structure of an image processing model provided by an embodiment of the present invention. Figure 2 As shown, the image processing model provided by the present invention includes a feature extraction network, a lightweight network converter, a multi-scale feature fusion network and an image reconstruction network, and these networks are connected in series in sequence.

[0058] In an embodiment of the present invention, a feature extraction network can be used to first extract features from an input image to be processed to obtain a first feature map. Then, the first feature map is processed in sequence by a network conversion module connected in series in a lightweight network converter to obtain a second feature map. The second feature map output by the lightweight network converter is input into a multi-scale feature fusion network for feature fusion processing to obtain a third feature map. The third feature map is reconstructed using an image reconstruction network to obtain a first image. It can be understood that the first image contains a target object to be filtered out. Finally, a subtraction operation is performed on the image to be processed and the first image to obtain a target image with the target object filtered out.

[0059] It should be noted that the image to be processed in the embodiment of the present invention may be a rainy image with rain streaks taken by a power monitoring device, an outdoor camera or other equipment, and the target object to be filtered out may be rain streaks, noise points and other objects in the rainy image that are likely to interfere with image recognition and image analysis. The target image is a rainless image with rain streaks and noise points removed.

[0060] As an example, a convolutional neural network with a convolution kernel size of 3 can be constructed as a feature extraction network to preliminarily extract the rain streak feature information to be filtered out from the input rainy image. Then, four series-connected LWNT structures (network conversion modules) are constructed as lightweight network converters, and a layer of convolution is set after the series-connected LWNT structure to increase translation invariance to obtain the second feature map corresponding to the image to be processed. It can be understood that translation is a geometric transformation, which means that every point in an image or a space is moved the same distance in the same direction. For example, for image classification tasks, no matter where the target in the image is moved to the picture, the result (label) should be the same. In neural networks, convolution is defined as a feature detector at different positions, which means that no matter where the target appears in the image, it will detect the same features and output the same response. After the image is translated, the expression on the corresponding feature map is also translated. For example, suppose that a rain streak area A in a rainy image is located in the upper left corner of the image. After convolution, the rain streak feature of the rain streak area A is also located in the upper left corner of the second feature map. Assuming that the rain streak area A in the rainy image is moved to the upper right corner, the convolution kernel will not detect the rain streak features of the rain streak area A until it moves to the upper right corner. Setting a layer of convolution after the series LWNT structure can ensure that the corresponding feature information can still be detected even if the image is translated, thereby increasing the translation invariance of the second feature map.

[0061] After obtaining the second feature map, a multi-scale feature fusion network with a lateral connection structure can also be constructed to fuse the multi-layer pyramid-shaped shallow position features and deep semantic features. The basic network structure of the multi-scale feature fusion network can use a residual network for residual mapping to obtain the third feature map corresponding to the image to be processed, thereby effectively transmitting the feature information further. It can be understood that the residual network is composed of multiple residual blocks stacked together, and a jump connection is used in the residual block. The residual map output by the residual block = the expected mapping output-the residual block input. Due to the change in learning objectives, the network no longer learns the complete expected mapping output H(x), but the residual mapping F(x) = H(x)-x of the output and input. In fact, it is easier to optimize the residual mapping than to optimize the original expected mapping. This short connection form does not introduce new parameters and does not increase the computational complexity. The input signal can be directly transmitted from any low layer to the high layer, which effectively solves the network degradation problem in the deep neural network, thereby transmitting the feature information further.

[0062] After obtaining the third feature map, a coordinate attention mechanism can be added to the image reconstruction network so that the image reconstruction network pays more attention to the area where the target object to be filtered out is located in the third feature map. Finally, a convolutional neural network with a convolution kernel size of 3 is used to reconstruct the image and output the first image containing the target object.

[0063] It should be noted that the input image to be processed is subjected to feature extraction through the feature extraction network and the lightweight network converter to obtain the second feature map, and then enters the multi-scale feature fusion network, and the third feature map is obtained through layer-by-layer calculation. Since there is downsampling between each layer of feature maps in the multi-scale feature hierarchical structure of the multi-scale feature fusion network, the multi-layer neural network will reduce the feature map resolution while enhancing the semantic information of the feature map, and the feature map hierarchy finally obtained is a pyramid-shaped structure. Although the top-level features of the feature pyramid experience more convolution times and have highly abstract semantic information, the receptive field of the top-level features is large, and it is not sensitive to the edge and position information of the target object, and the detail feature information is difficult to be represented. Such a low resolution is far from enough to remove the target object in the image to be processed, and further detail information needs to be added. The bottom-level features of the feature pyramid experience fewer convolution times and their receptive fields are also smaller, which can help the network effectively focus on detail information. Therefore, in an embodiment of the present invention, a top-down feature pyramid fusion structure with horizontal connections is constructed as a multi-scale feature fusion network, and the detail information concerned by the bottom-level features is combined with the global information concerned by the top-level features to obtain complete target object information.

[0064] Reference Figure 3 , which shows a schematic diagram of the structure of a multi-scale feature fusion network provided by the present invention. Figure 3As shown, the present invention constructs a top-down feature pyramid fusion structure with lateral connections as a multi-scale feature fusion network based on a bottom-up feedforward neural network structure.

[0065] For the input of the multi-scale feature fusion network (the second feature map, size C×H×W), first use two dilated convolutions (Dilated Conv) with a size of 3 and dilation factors of 1 and 2 respectively to extract feature information with receptive field scales of 3×3 and 5×5, and use the activation function (Leaky Relu). Then, the features of different scales are spliced ​​through "concatenation + convolution" (Concat+Conv), and 1×1 convolution is used for channel dimensionality reduction. The rain streak feature k after dimensionality reduction can be expressed as:

[0066] k=COnV,x,[@(COnV:x:,(W)),cp(ConV_:x@;i(n))J( I )

[0067] Among them, i×i，j Represents a dilated convolution with a convolution kernel size of i and a dilation coefficient of j; Conv 1×1 Represents a 1×1 convolution; (Conv()) represents an activation function; n is the second feature map in the embodiment of the present invention.

[0068] In order to extract the scale features of rain streaks at a fine-grained level, the rain streak feature k is extracted using a multi-level segmentation convolution splicing method in the channel dimension. First, k is split into four features M2, M3, M4, and M5 with the same number of channels, and the size of each feature map is (C / 4×H×W). An additional 3×3 convolution layer is added after the M feature layer to obtain the final P5, P4, P3, and P2 layer features. Specifically, the first group of feature maps M2 undergoes convolution (Conv) with a convolution kernel size of 3, and the output feature is P2. P2 is split in half in the channel dimension into P1,1 and P1,2, and P1,2 is spliced ​​with the next group of features M3 through "connection + convolution" (Concat+Conv) and sent to the convolution with a convolution kernel size of 3. This operation is repeated three times, so that the spliced ​​features are fully reused. The features in the previous group have undergone fewer convolutions and smaller receptive fields, which can help the network effectively focus on detail information. The features contained in the latter groups undergo more convolutions, and their receptive fields are larger, which can help the network focus on global information.

[0069] In addition, a P6 layer is specially added to handle larger scales. It is obtained by downsampling P5. The final {P2, P3, P4, P5, P6} feature pyramid map is transmitted to the recovery network module to concatenate P2, P3, P4, P5, P6 along the channel dimension, and then use 1×1 convolution to reduce the channel dimension.

[0070] The embodiment of the present invention uses a top-down multi-scale feature pyramid structure to merge the bottom and top feature maps to generate a feature map with both high resolution and rich semantic information, fully learn important target object spatial regions, more completely obtain target object information, and improve the ability to filter out target objects in images.

[0071] Reference Figure 4 , which shows a schematic diagram of the structure of an image reconstruction network provided by the present invention. Figure 4 As shown, the present invention introduces a coordinate attention mechanism in the image reconstruction network, and embeds position information into the channel attention, thereby obtaining the object information of the global target with a small network overhead. The embodiment of the present invention further enhances the performance of the image processing model by continuing to construct the coordinate attention mechanism. The coordinate attention first uses pooling kernels of sizes (H, 1) and (1, W) to use global average pooling (X Avg Pooling) for each channel along the horizontal and vertical coordinates of the input third feature map y (size C×H×W). Therefore, the output of the cth channel with a height of h and the output of the cth channel with a width of w can be expressed as:

[0072]

[0073] in, They represent the components with coordinates (h, i) and (j, w) and channel dimension c in the third feature map y of the input respectively; Represents the output component of the cth channel with height h; represents the output component of the cth channel with a width of w. Then z is convolved by “Concat Conv” h With z w Splicing is performed on the channel dimension, and a shared 1×1 convolution is used for channel dimension reduction, followed by the H-swish nonlinear activation function (Non-Linear) to obtain the intermediate feature t:

[0074] t=δ(Conv 1×1 (z h , z w )) (3)

[0075] Among them, Conv 1×1represents a 1×1 convolution; δ(Conv()) represents a nonlinear activation function.

[0076] Split t into two independent components z along the space h and z w , and use two 1×1 convolutions (Conv) to convert the two independent components into components with the same number of channels as the input feature y, and then use the Sigmoid activation function, which can be expressed as:

[0077] m h =σ(Conv 1×1 (t h )),

[0078] m w =σ(Conv 1×1 (t w )), (4)

[0079] Among them, m h and m w Respectively represent the coordinate attention weights on the two coordinate axes, and σ(Conv()) represents the activation function. The output of coordinate attention can be expressed as:

[0080]

[0081] Among them, u c (i, j) means the coordinates in the output third feature map are (i, j), through the component of dimension c; y c (i, j) represents the component with coordinate (i, j) and channel dimension c in the third feature map y of the input; Represents the coordinate attention weight of the c-th channel with height h on the horizontal coordinate axis, Represents the coordinate attention weight of the c-th channel with width w on the vertical axis,

[0082] The first image output by the image reconstruction network can be expressed as:

[0083] M=n+u(6)

[0084] By constructing a position attention mechanism, not only can cross-channel information be captured, but also position and direction information can be perceived, so that the model can more accurately locate and identify the target area, refine the feature extraction process, and improve the accuracy of removing target objects.

[0085] In an optional embodiment of the present invention, the lightweight network converter further includes a first convolutional layer; the step of inputting the first feature map corresponding to the image to be processed into the lightweight network converter of the image processing model for feature extraction includes:

[0086] Step S21, using each of the network conversion modules to process the first feature map in turn to obtain the second feature map;

[0087] Step S22: Use the first convolution layer to perform convolution processing on the second feature map to obtain a first convolution result.

[0088] Among them, the output data of the lightweight network converter is a residual mapping of the input data of the lightweight network converter and the first convolution result output by the first convolution layer.

[0089] In an embodiment of the present invention, a lightweight network converter can be constructed by multiple network conversion modules connected in series, and a first convolution layer can be connected in series after the last network conversion module. The convolution layer can be composed of a convolutional neural network with a convolution kernel size of 3. In an embodiment of the present invention, the original input of the first convolution layer, that is, the input data of the lightweight network converter, is added to the first convolution result output by the first convolution layer during residual mapping by using residual design, so as to obtain a result (output data of the lightweight network converter) without changing the feature scale. It should be noted that the residual block used to implement residual mapping has two convolution layers with the same number of output channels, each convolution layer is followed by a batch normalization layer and a ReLU activation function, and then through the cross-layer data path, these two convolution operations are skipped, and the input of the residual block (that is, the input data of the lightweight network device) is directly added before the last ReLU activation function, which requires that the output of the two convolution layers has the same shape as the input, so that they can be added. Therefore, the output data obtained after residual mapping, that is, the output data X of the lightweight network converter out , which is the same shape as the original input data, that is, the input data X0 of the lightweight network converter. In other words, X0 is the same as X out The characteristic scale is the same.

[0090] For the input data X0 of the lightweight network converter, the output data X0 of the lightweight network converter obtained by residual mapping with the first convolution result is out It can be expressed as:

[0091] X out =C(X6)+X0 (7)

[0092] Among them, C(X6) represents the first convolution result of the convolution layer.

[0093] The lightweight network converter in the embodiment of the present invention not only increases the translation invariance through the convolution layer, but also further refines the global features of the processed image through multiple serially connected network conversion modules (LWNT structure).

[0094] Optionally, the network conversion module includes a self-attention module and a multi-layer perceptron; the input data of the multi-layer perceptron is a residual mapping of the output data of the self-attention module and the input data of the network conversion module. When performing residual mapping, the output data output by the self-attention module is added to the input data of the network conversion module to obtain the input data of the multi-layer perceptron.

[0095] In an embodiment of the present invention, each network conversion module of the lightweight network encoder may include a region based self-attention module (RBSA) and a multi-layer perceptron (MLP), which effectively solves the network degradation problem in deep neural networks through residual connections and ensures the stability of the structure.

[0096] Optionally, the input ends of the self-attention module and the multi-layer perceptron each include a normalization layer to normalize the input data of the self-attention module and the multi-layer perceptron respectively, map the data to a range of 0 to 1 for processing, and improve the processing speed of the self-attention module and the multi-layer perceptron.

[0097] Reference Figure 5 , shows a schematic diagram of the structure of a network conversion module provided by the present invention, such as Figure 5 As shown, the input end of the self-attention module RBSA and the multi-layer perceptron MLP both include a normalization layer LN, and the self-attention module RBSA and the multi-layer perceptron MLP are connected via a residual. i-1 , and the output data of the self-attention module RBSA are added, and the result is used as the input data of the normalization layer of the multi-layer perceptron MLP . Compare the output data of the multi-layer perceptron MLP with the input data of the normalization layer of the multi-layer perceptron MLP By adding, we can get the output data z of the network conversion module i .

[0098] It should be noted that the self-attention module (Self-Attention) uses an attention mechanism that associates different positions of a single sequence to calculate the representation of the same sequence to process the input data (the output data of the normalization layer of the self-attention module), establish global dependencies, and expand the receptive field of the feature map to obtain more context information. The embodiment of the present invention does not limit the specific structure and number of network layers of the self-attention module. Exemplarily, the self-attention module in the embodiment of the present invention can be based on the regional self-attention mechanism module RBSA, which captures source-process dependencies by introducing regional growth self-attention that can utilize the local context information of neighborhood points, and is suitable for a variable number of objects, such as rain streaks to be filtered out in a rainy image, thereby further refining the global features of the image.

[0099] Multilayer perceptron MLP is a deep learning model based on feedforward neural network. In addition to the input layer and the output layer, the multilayer perceptron in the embodiment of the present invention also includes multiple hidden layers. The embodiment of the present invention does not limit the number of hidden layers. Exemplarily, a multilayer perceptron including two hidden layers can be used. The hidden layer is fully connected to the input layer. Assuming that the input layer is represented by a vector X, the input of the hidden layer connected to the input layer is f(W1X+b1), W1 is the weight (also called the connection coefficient), b1 is the bias, and the function can be a sigmoid function or a tanh function. Each neuron in the hidden layer receives the output of the previous layer, performs a weighted sum on it, and processes it through an activation function to generate the output of the hidden layer. The outputs of these hidden layers are then passed to the next layer until the output layer is reached. The hidden layer to the output layer is a classifier softmax regression. In the embodiment of the present invention, the characteristic information of the target object can be automatically learned by the multilayer perceptron, and the MLP can handle polyphenol problems, has good generalization ability, and is conducive to improving the generalization ability of the LWNT structure in the present invention.

[0100] Reference Figure 6 , shows a schematic diagram of the structure of a lightweight network converter provided by the present invention. Figure 6 As shown in FIG. , the lightweight network encoder consists of 4 serially connected Figure 5 The network conversion module shown in FIG. 1 is composed of the output of the current network conversion module as the input of the next network conversion module. Each network conversion module includes a self-attention module and a multi-layer perceptron. The input ends of the self-attention module and the multi-layer perceptron each include a normalization layer to normalize the input data of the self-attention module and the multi-layer perceptron respectively, and the residual connection is used to ensure the stability of the structure. In addition, the last network conversion module can be connected in series with a first convolutional layer ( Figure 6(not shown in the figure), which is used to increase the translation invariance of the lightweight network converter. The output data of the lightweight network converter is the residual mapping of the input data of the lightweight network converter (that is, the input data of the first network conversion module) and the first convolution result output by the first convolution layer.

[0101] In an optional embodiment of the present invention, the feature extraction network includes a second convolutional layer and a residual block; the step of inputting the image to be processed into the feature extraction network for feature extraction to obtain a first feature map includes:

[0102] Step S31, using the second convolution layer to perform convolution processing on the image to be processed to obtain a second convolution result;

[0103] Step S32: Calculate the residual map of the second convolution result and the image to be processed by using the residual block to obtain a first feature map.

[0104] It should be noted that in traditional convolutional neural networks, the network is relatively shallow, but the number of parameters is often very large. Usually, a deeper network can extract more essential features from aerial images, and the performance of the network can be improved by increasing the width and depth of the network. However, due to the existence of network degradation, deep neural network training is very difficult, but the use of residual networks can reduce the difficulty of deep network training while increasing the number of network layers.

[0105] In an embodiment of the present invention, a feature extraction network can be constructed based on a residual network to train a deep neural network. Figure 7 , shows a schematic diagram of the structure of a residual network. The core of the residual network is the residual jump connection mechanism, such as Figure 7 As shown, x is the input of the residual block (that is, the input of the second convolutional layer in the present invention: the image to be processed), H(x) is the expected mapping output (that is, the second convolution result output by the second convolutional layer in the present invention), then the first feature map output by the residual block is the residual mapping F(x) = H(x) - x. Among them, relu refers to the activation function (Rectified linear unit, ReLU). The x identity is the identity mapping. For any set A, if the mapping f:A→A is defined as f(x) = x, that is, it is stipulated that each element x in A corresponds to itself, then f is called the identity mapping on A. The weight layer is a function of the weight w' and the input x, which can be written as F'(x, w'). The ultimate goal of training the residual block is to find a set of weights w' so that F'(x, w') is closest to the true function H(x). However, the real function H(x) is often very complex, and it is difficult to find a sufficiently close function F'(x, w'), so the identity mapping is introduced. Figure 7As shown in the figure, the input of the residual block is still x, and the input x is superimposed once before the output. The network structure has changed, w' becomes w, and F'(x, w') becomes F(x, w). The training purpose has also changed, and it has become to find a set of weights w so that F(x, w)+x→H(x), that is, F(x, w)→H(x)-x. Due to the change in learning objectives, the network no longer learns the complete output H(x), but the residual mapping H(x)-x of the output and input. In fact, optimizing the residual mapping is easier than optimizing the original mapping. This short connection form does not introduce new parameters and does not increase the computational complexity. The feature information x is obtained through jump connections (that is, Figure 7 The skip connection, also known as the shortcut connection, directly skips the input x and adds it to the output without adding any additional parameters or increasing the complexity of network calculations. It greatly reduces the repetitiveness of feature extraction and effectively solves the problem of network performance degradation caused by the increase in the number of layers.

[0106] As an example, in the application scenario of image deraining, the ResNet-50 residual network can be selected as the feature extraction network in the present invention to extract image features according to the requirements of rain streak characteristics, density, and model size in the image deraining task. The network uses convolution with a step size of 2 instead of pooling for downsampling, and constructs a small convolution kernel bottleneck structure such as 1×1 and 3×3. Two 1×1 convolutions reduce the dimension of the number of channels first and then restore the dimension, while introducing more nonlinear mappings, greatly reducing the amount of calculation.

[0107] In an optional embodiment of the present invention, before using a pre-trained image processing model to process the image to be processed to obtain a target image from which the target object is filtered out, the method further includes:

[0108] Step S41, obtaining a training sample set; the training sample set includes a sample image and a label image corresponding to the sample image;

[0109] Step S42, inputting the sample image into the candidate image processing model for processing to obtain an output image;

[0110] Step S43, calculating a first loss value of the image processing model based on a combined mean square error of the output image and the label image;

[0111] Step S44, calculating a second loss value of the image processing model based on the structural similarity between the output image and the label image;

[0112] Step S45: adjust the model parameters of the candidate image processing model according to the first loss value and the second loss value and perform the next round of training until the preset conditions are met to obtain the image processing model.

[0113] In an embodiment of the present invention, a hybrid loss function combining mean square error (MSE) and structural similarity (SSIM) can be used to optimize the image processing model. MSE is widely used in the task of removing rain streaks from images, and SSIM is used to measure the structural similarity between two images, which can make the image after rain removal closer to human vision and clearer. The loss function is defined as follows:

[0114]

[0115] L SSIM = 1-SSIM(h(x), I), (8)

[0116] Where N is the number of samples, x is the sample image input to the image processing model, h(x) is the output image of the image processing model, I is the label image that does not contain the target object, and SSIM(h(x), I) is the structural similarity between the output image of the image processing model and the label image. MSE and L SSIM , the final loss function L can be L MSE and L SSIM sum:

[0117] L=L MSE +L SSIM (9)

[0118] During the training process, the model parameters can be adjusted based on the target loss value calculated by the above formula (9) until the training termination condition is met to obtain a trained image processing model. It can be understood that the training termination condition can be that the number of training times is greater than a preset number threshold, or the target loss value is less than a preset loss value, or the error between the target loss values ​​obtained from consecutive rounds of training is less than a preset value, etc.

[0119] For example, when performing the image deraining task using the image processing method provided by the embodiment of the present invention, in order to verify the effectiveness and generalization of the algorithm, 2000 pairs of transmission line machine inspection images were collected, in which the training set and the test set were allocated in a ratio of 8:2, and the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used as quantitative evaluation indicators to measure the performance of the image processing model provided by the present invention. The data set has dense rain lines, so it is more challenging and can better test the effectiveness of the algorithm.

[0120] In an embodiment of the present invention, by constructing a ResNet-50 feature extraction residual network, the difficulty of deep network training is reduced and the amount of neural network calculation is reduced. On the basis of the bottom-up feature extraction network, a top-down multi-scale feature pyramid fusion structure with lateral connections is constructed to combine low-level position information with high-level semantic information to fully learn comprehensive rain streak information. Secondly, a coordinate attention mechanism is introduced to perceive position and direction information so that the model can more accurately locate and identify the target area. In order to better study the feasibility of the method of the present invention for rain removal of drone patrol images, the effects of multi-scale fusion and coordinate attention mechanism on image rain removal are first explored. Except for these two parameters, other parameters and training steps remain unchanged. The image processing model proposed in the present invention is used as the backbone framework. The evaluation results are shown in Table 1.

[0121] Table 1

[0122] Feature Fusion Coordinate Attention PSNR / dB SSIM no no 26.12 0.785 yes no 27.87 0.823 no yes 28.51 0.881 yes yes 29.61 0.903

[0123] As can be seen from Table 1, after adding a multi-scale structure to the image processing model, the PSNR and SSIM results are improved, with PSNR and SSIM increased by 1.75 and 0.038 respectively. This shows the effectiveness of enhancing the expression of position and semantic information through a multi-scale feature fusion strategy for image deraining, making the model robust to global rain streak information. The image processing model of the present invention also obtains a PSNR improvement by using coordinate attention, which reflects that coordinate attention can improve the accuracy of rain removal details and further refine the feature extraction process. The experiment fully proves that the use of feature fusion and coordinate attention can effectively improve the model PSNR and SSIM, which finally reached 29.61 and 0.903 respectively.

[0124] In order to further verify the effectiveness of the image processing method provided by the present invention, five different target detection algorithms, including JORDER, DRN, PReNet and the algorithm proposed by the present invention (using VGG-16 and ResNet-50 as feature extraction frameworks respectively), are used for experimental comparison. The evaluation results are shown in Table 2.

[0125] Table 2

[0126]

[0127]

[0128] As can be seen from Table 2, the overall performance of the image processing method provided by the embodiment of the present invention is better than other deep learning rain removal algorithms. After a series of improvements such as introducing the ResNet-50 feature extraction network, building a multi-scale feature fusion structure, and adopting coordinate attention, the image processing method provided by the present invention has an increase of 5.06 PSNR and 0.068 SSIM compared to the original JORDER. The evaluation indicators of the image processing model provided by the present invention are better than other target detection comparison algorithms, which fully demonstrates the advanced nature of the image processing method provided by the present invention for image rain removal.

[0129] In summary, the embodiments of the present invention provide an image processing method, which converts the image to be processed into a feature space by utilizing the feature extraction network in the image processing model, thereby replacing the feature extraction method based on prior information in the related art. In addition, the embodiments of the present invention refine the global image features corresponding to a single image through the network conversion modules connected in series in the lightweight network converter, and fuses the position details and spatial context information of the image through the multi-scale feature fusion network, thereby avoiding the feature loss of target objects such as rain streaks, watermarks, noise points to be filtered out in a single image, and can generate a clearer target image, thereby improving the image processing effect.

[0130] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0131] Device Embodiment

[0132] Reference Figure 8 , shows a structural block diagram of an embodiment of an image processing device of the present invention, and the device may specifically include:

[0133] An image acquisition module 801 is used to acquire an image to be processed, wherein the image to be processed contains a target object that needs to be removed in a preset image processing model;

[0134] A first extraction module 802 is used to input the first feature map corresponding to the image to be processed into the lightweight network converter of the image processing model for feature extraction to obtain a second feature map;

[0135] A fusion and reconstruction module 803 is used to input the second feature map into the multi-scale feature fusion network of the image processing model for feature fusion, and obtain a first image after image reconstruction;

[0136] An image processing module 804 is used to process the image to be processed using the first image to obtain a target image from which the target object is filtered out;

[0137] The lightweight network converter includes at least two network conversion modules connected in series for refining shallow features of the image to be processed.

[0138] Optionally, the image processing model further includes a feature extraction network, wherein the feature extraction network, the lightweight network converter and the multi-scale feature fusion network are sequentially connected in series; and the device further includes:

[0139] The second extraction module is used to input the image to be processed into the feature extraction network for feature extraction to obtain the first feature map; the first feature map is used to reflect the shallow features of the image to be processed.

[0140] Optionally, the image processing model further includes an image reconstruction network; the fusion reconstruction module includes:

[0141] A feature fusion submodule, used for inputting the second feature map into the multi-scale feature fusion network for feature fusion to obtain a third feature map;

[0142] The image reconstruction submodule is used to input the third feature map into the image reconstruction network to perform image reconstruction to obtain a first image.

[0143] The image processing module comprises:

[0144] The subtraction operation submodule is used to perform a subtraction operation on the image to be processed and the first image to obtain a target image.

[0145] Optionally, the lightweight network converter further includes a first convolutional layer; and the first extraction module includes:

[0146] A conversion processing unit, used to process the first feature map in sequence using each of the network conversion modules to obtain the second feature map; the second feature map is used to reflect the shallow position features and deep semantic features of the image to be processed;

[0147] A convolution unit, configured to perform convolution processing on the second feature map using the first convolution layer to obtain a first convolution result;

[0148] Among them, the output data of the lightweight network converter is a residual mapping of the input data of the lightweight network converter and the first convolution result output by the first convolution layer.

[0149] Optionally, the network conversion module includes a self-attention module and a multi-layer perceptron; the input data of the multi-layer perceptron is a residual mapping of the output data of the self-attention module and the input data of the network conversion module.

[0150] Optionally, the input ends of the self-attention module and the multilayer perceptron each include a normalization layer to normalize the input data.

[0151] Optionally, the feature extraction network includes a second convolutional layer and a residual block; the second extraction module includes:

[0152] A convolution unit, configured to perform convolution processing on the image to be processed by using the second convolution layer to obtain a second convolution result;

[0153] A residual mapping unit is used to calculate the residual mapping of the second convolution result and the image to be processed using the residual block to obtain a first feature map.

[0154] Optionally, the device further comprises:

[0155] A training sample set acquisition module, used to acquire a training sample set; the training sample set includes a sample image and a label image corresponding to the sample image;

[0156] An input module, used for inputting the sample image into a candidate image processing model for processing to obtain an output image;

[0157] A first calculation module, configured to calculate a first loss value based on a combined mean square error of the output image and the label image;

[0158] A second calculation module, used for calculating a second loss value based on the structural similarity between the output image and the label image;

[0159] A parameter adjustment module is used to adjust the model parameters of the candidate image processing model according to the first loss value and the second loss value and perform the next round of training until the preset conditions are met to obtain the image processing model.

[0160] In summary, an embodiment of the present invention provides an image processing device, which converts an image to be processed into a feature space by utilizing a feature extraction network in an image processing model, thereby replacing a feature extraction method based on prior information in related technologies. In addition, an embodiment of the present invention extracts global image features corresponding to a single image through a network conversion module connected in series in a lightweight network converter, and fuses the position details and spatial context information of the image through a multi-scale feature fusion network, thereby avoiding feature loss of target objects such as rain streaks, watermarks, and noise points to be filtered out in a single image, improving the accuracy of the filtering results, being able to generate clearer target images, and improving the image processing effect.

[0161] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0162] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0163] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0164] An embodiment of the present invention provides an electronic device for image processing, the electronic device comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing the following operations:

[0165] A11, obtaining an image to be processed, wherein the image to be processed contains a target object that needs to be removed in a preset image processing model;

[0166] A12, inputting the first feature map corresponding to the image to be processed into the lightweight network converter of the image processing model for feature extraction to obtain a second feature map;

[0167] A13, inputting the second feature map into the multi-scale feature fusion network of the image processing model to perform feature fusion, and obtaining a first image after image reconstruction;

[0168] A14, using the first image to process the image to be processed to obtain a target image from which the target object is filtered out;

[0169] The lightweight network converter includes at least two network conversion modules connected in series for refining shallow features of the image to be processed.

[0170] Optionally, the image processing model further includes a feature extraction network, wherein the feature extraction network, the lightweight network converter and the multi-scale feature fusion network are sequentially connected in series; and the electronic device is further configured to execute the one or more programs by one or more processors, including instructions for performing the following operations:

[0171] The image to be processed is input into the feature extraction network for feature extraction to obtain the first feature map; the first feature map is used to reflect the shallow features of the image to be processed.

[0172] Optionally, the image processing model further includes an image reconstruction network; the step of inputting the second feature map into a multi-scale feature fusion network of the image processing model for feature fusion, and obtaining the first image after image reconstruction includes:

[0173] Inputting the second feature map into the multi-scale feature fusion network for feature fusion to obtain a third feature map;

[0174] The third feature map is input into the image reconstruction network to perform image reconstruction to obtain the first image.

[0175] Optionally, the using the first image to process the image to be processed to obtain a target image from which the target object is filtered out includes:

[0176] A subtraction operation is performed on the image to be processed and the first image to obtain the target image.

[0177] Optionally, the lightweight network converter further includes a first convolutional layer; and the step of inputting the first feature map corresponding to the image to be processed into the lightweight network converter of the image processing model for feature extraction includes:

[0178] Using each of the network conversion modules to process the first feature map in turn to obtain the second feature map; the second feature map is used to reflect the shallow position features and deep semantic features of the image to be processed;

[0179] Using the first convolution layer to perform convolution processing on the second feature map to obtain a first convolution result;

[0180] Among them, the output data of the lightweight network converter is a residual mapping of the input data of the lightweight network converter and the first convolution result output by the first convolution layer.

[0181] Optionally, the network conversion module includes a self-attention module and a multi-layer perceptron; the input data of the multi-layer perceptron is a residual mapping of the output data of the self-attention module and the input data of the network conversion module.

[0182] Optionally, the input ends of the self-attention module and the multilayer perceptron each include a normalization layer to normalize the input data.

[0183] Optionally, the feature extraction network includes a second convolutional layer and a residual block; and inputting the image to be processed into the feature extraction network for feature extraction includes:

[0184] Using the second convolution layer to perform convolution processing on the image to be processed to obtain a second convolution result;

[0185] The residual block is used to calculate a residual map of the second convolution result and the image to be processed to obtain a first feature map.

[0186] Optionally, the electronic device is further configured to execute, by one or more processors, the one or more programs including instructions for performing the following operations:

[0187] Acquire a training sample set; the training sample set includes a sample image and a label image corresponding to the sample image;

[0188] Inputting the sample image into a candidate image processing model for processing to obtain an output image;

[0189] Calculating a first loss value based on a combined mean square error of the output image and the label image;

[0190] Calculating a second loss value based on the structural similarity between the output image and the label image;

[0191] The model parameters of the candidate image processing model are adjusted according to the first loss value and the second loss value, and the next round of training is performed until the preset conditions are met to obtain the image processing model.

[0192] Fig. 9 1 is a block diagram of an electronic device 600 for image processing according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0193] Reference Fig. 9 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0194] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0195] The memory 604 is configured to store various types of data to support operations on the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, videos, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0196] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0197] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0198] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice information processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0199] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0200] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of user contact with the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0201] The communication component 616 is configured to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0202] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0203] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0204] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device (server or terminal), enables the processor to execute Figure 1 The image processing method shown.

[0205] The image processing method, device and electronic device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed; the image to be processed contains a target object that needs to be removed in a preset image processing model; Inputting the first feature map corresponding to the image to be processed into the lightweight network converter of the image processing model for feature extraction to obtain a second feature map; Inputting the second feature map into the multi-scale feature fusion network of the image processing model for feature fusion, and obtaining a first image after image reconstruction; Processing the image to be processed using the first image to obtain a target image from which the target object is filtered out; The lightweight network converter includes at least two network conversion modules connected in series for refining shallow features of the image to be processed.

2. The method according to claim 1, characterized in that The image processing model further includes a feature extraction network, wherein the feature extraction network, the lightweight network converter and the multi-scale feature fusion network are sequentially connected in series; the method further includes: The image to be processed is input into the feature extraction network for feature extraction to obtain the first feature map; the first feature map is used to reflect the shallow features of the image to be processed.

3. The method according to claim 2, characterized in that The image processing model further includes an image reconstruction network; the step of inputting the second feature map into the multi-scale feature fusion network of the image processing model for feature fusion, and obtaining the first image after image reconstruction, includes: Inputting the second feature map into the multi-scale feature fusion network for feature fusion to obtain a third feature map; The third feature map is input into the image reconstruction network to perform image reconstruction to obtain the first image.

4. The method according to claim 1, characterized in that The lightweight network converter further includes a first convolutional layer; the step of inputting the first feature map corresponding to the image to be processed into the lightweight network converter of the image processing model for feature extraction includes: Using each of the network conversion modules to process the first feature map in turn to obtain the second feature map; the second feature map is used to reflect the shallow position features and deep semantic features of the image to be processed; Using the first convolution layer to perform convolution processing on the second feature map to obtain a first convolution result; Among them, the output data of the lightweight network converter is a residual mapping of the input data of the lightweight network converter and the first convolution result output by the first convolution layer.

5. The method according to claim 4, characterized in that The network conversion module includes a self-attention module and a multi-layer perceptron; the input data of the multi-layer perceptron is a residual mapping of the output data of the self-attention module and the input data of the network conversion module.

6. The method according to claim 5, characterized in that The input ends of the self-attention module and the multi-layer perceptron both include a normalization layer to normalize the input data.

7. The method according to claim 2, characterized in that The feature extraction network includes a second convolutional layer and a residual block; the step of inputting the image to be processed into the feature extraction network for feature extraction includes: Using the second convolution layer to perform convolution processing on the image to be processed to obtain a second convolution result; The residual block is used to calculate a residual map of the second convolution result and the image to be processed to obtain a first feature map.

8. The method according to claim 1, characterized in that The step of processing the image to be processed by using the first image to obtain a target image from which the target object is filtered out includes: A subtraction operation is performed on the image to be processed and the first image to obtain the target image.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Acquire a training sample set; the training sample set includes a sample image and a label image corresponding to the sample image; Inputting the sample image into a candidate image processing model for processing to obtain an output image; Calculating a first loss value based on a combined mean square error of the output image and the label image; Calculating a second loss value based on the structural similarity between the output image and the label image; The model parameters of the candidate image processing model are adjusted according to the first loss value and the second loss value, and the next round of training is performed until the preset conditions are met to obtain the image processing model.

10. An image processing device, characterized in that: The device comprises: An image acquisition module is used to acquire an image to be processed, wherein the image to be processed contains a target object that needs to be removed in a preset image processing model; A first extraction module, used for inputting a first feature map corresponding to the image to be processed into a lightweight network converter of the image processing model for feature extraction to obtain a second feature map; A fusion and reconstruction module, used for inputting the second feature map into the multi-scale feature fusion network of the image processing model for feature fusion, and obtaining a first image after image reconstruction; An image processing module, used to process the image to be processed using the first image to obtain a target image from which the target object is filtered out; The lightweight network converter includes at least two network conversion modules connected in series for refining shallow features of the image to be processed.

11. The device according to claim 10, characterized in that The image processing model further includes a feature extraction network, wherein the feature extraction network, the lightweight network converter and the multi-scale feature fusion network are sequentially connected in series; the device further includes: The second extraction module is used to input the image to be processed into the feature extraction network for feature extraction to obtain the first feature map; the first feature map is used to reflect the shallow features of the image to be processed.

12. The device according to claim 11, characterized in that The image processing model also includes an image reconstruction network; the fusion reconstruction module includes: A feature fusion submodule, used for inputting the second feature map into the multi-scale feature fusion network for feature fusion to obtain a third feature map; The image reconstruction submodule is used to input the third feature map into the image reconstruction network to perform image reconstruction to obtain a first image.

13. The device according to claim 10, characterized in that The lightweight network converter further includes a first convolutional layer; the first extraction module includes: A conversion processing unit, used to process the first feature map in sequence using each of the network conversion modules to obtain the second feature map; the second feature map is used to reflect the shallow position features and deep semantic features of the image to be processed; A convolution unit, configured to perform convolution processing on the second feature map using the first convolution layer to obtain a first convolution result; Among them, the output data of the lightweight network converter is a residual mapping of the input data of the lightweight network converter and the first convolution result output by the first convolution layer.

14. An electronic device, characterized in that: The electronic device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to execute the image processing method according to any one of claims 1 to 9 by one or more processors.

15. A readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the processor is enabled to execute the image processing method as claimed in any one of claims 1 to 9.

Images