Image processing method, device and equipment

Abstract

The invention provides an image processing method, device and equipment, which can be applied to the technical field of image processing. The image processing method comprises the following steps: preprocessing an input image to obtain input features; respectively inputting the input features into a visual encoder and a multi-layer perceptron in a visual center decoupler to obtain enhanced special features and saliency target detection special features; the visual encoder aggregates local area features based on the input features to obtain enhanced special features, and the multi-layer perceptron captures edge information based on the input features to obtain saliency target detection special features; inputting the enhanced special features into an enhanced network to obtain enhanced output features; the enhancement network takes the illumination weights of different color channels and the local binary pattern features of the input image as illumination constraints, and enhances the special enhancement features to obtain enhanced output features; and inputting the saliency target detection special feature and the enhanced output feature into a saliency target detection network to detect a saliency target.

Description

TECHNICAL FIELD

[0001] At least one embodiment of the present application relates to the technical field of image segmentation, and more particularly to an image processing method, device and equipment. BACKGROUND

[0002] The core goal of salient object detection (SOD) is to identify and locate important and prominent regions in images, helping computer vision systems quickly locate and recognize key targets in complex scenes, thereby achieving more efficient image understanding and analysis. However, images captured in low-light environments generally have low contrast, poor target-background differentiation, and other problems, limiting the development of salient object detection technology.

[0003] Salient object detection models mainly use techniques such as pooling, downsampling, multi-scale, and integrity learning to achieve more effective feature fusion, thereby improving the performance of salient object detection models. However, low-light scenes are ubiquitous and unavoidable in daily life, and most salient object detection models simply fuse foreground and background features when processing low-light data, without effectively mitigating the interference of dark degradation factors on foreground-background segmentation, resulting in low salient object detection performance. SUMMARY

[0004] In view of the above problems, the present application provides an image processing method, device and equipment for improving the performance of salient object detection.

[0005] According to a first aspect of the present application, an image processing method is provided, which comprises: pre-processing an input image to obtain input features; inputting the input features into a visual encoder and a multi-layer perception machine in a visual center decoupler to obtain enhanced specialized features and salient object detection specialized features; the visual encoder is used to aggregate local region features based on the input features to obtain the enhanced specialized features, and the multi-layer perception machine is used to capture edge information based on the input features to obtain the salient object detection specialized features; inputting the enhanced specialized features into an enhancement network to obtain enhanced output features; wherein the enhancement network uses illumination weights of different color channels and local binary pattern features of the input image as illumination constraints to enhance the enhanced specialized features to obtain the enhanced output features; and inputting the salient object detection specialized features and the enhanced output features into a salient object detection network to detect salient objects.

[0006] According to an embodiment of the present application, the input features are input into the visual encoder and the multi-layer perception machine in the visual center decoupler to obtain the enhanced special features and the salient object detection special features, including: inputting the input features into the visual encoder to obtain first output features; inputting the input features into the multi-layer perception machine to obtain second output features; splicing the first output features and the second output features to obtain third output features; decoupling the third output features by using a first convolutional network to obtain the enhanced special features; decoupling the third output features by using a second convolutional network to obtain the salient object detection special features.

[0007] According to an embodiment of the present application, the input features are input into the visual encoder to obtain the first output features, including: mapping the input features into a plurality of pixel features pixel by pixel to obtain a first input set; encoding the first input set based on a visual center codebook to obtain a plurality of first position features corresponding to the plurality of visual centers in the visual center codebook; fusing the plurality of first position features corresponding to the plurality of visual centers to obtain second position features; multiplying the first input set and the second position features at a channel level to obtain the local region features; adding the first input set and the local region features at a channel level to obtain the first output features.

[0008] According to an embodiment of the present application, the input features are input into the multi-layer perception machine to obtain the second output features, including: grouping the input features along a channel dimension to obtain a second input set; inputting the second input set into a deep convolution unit to perform independent convolution operations on the features of each channel of the second input set to obtain a second input set after convolution; inputting the second input set after convolution into a channel multi-layer perception machine unit to fuse the features of each channel of the second input set after convolution to obtain the second output features.

[0009] According to an embodiment of the present application, the enhanced special features are input into the enhancement network to obtain enhanced output features, including: inverting each color channel of the enhanced special features; applying an illumination weight to each inverted color channel and splicing to obtain an attention weight map; extracting texture features of the input image based on local binary pattern features of the input image; splicing the attention weight map and the texture features to obtain the enhanced output features.

[0010] According to an embodiment of the present application, the salient object detection network comprises G dynamic multi-scale convolutions; and the inputting of the salient object detection special feature and the enhanced output feature into the salient object detection network for detecting a salient object comprises: inputting the salient object detection special feature into the first dynamic multi-scale convolution to obtain a first rotation feature; inputting the (i-1)th rotation feature and the enhanced output feature into the ith dynamic multi-scale convolution to obtain an ith rotation feature; wherein i is an integer greater than 1 and less than or equal to G, and the Gth rotation feature is the output of the salient object detection network.

[0011] According to an embodiment of the present application, the inputting of the (i-1)th rotation feature and the enhanced output feature into the ith dynamic multi-scale convolution to obtain the ith rotation feature comprises: splicing the (i-1)th rotation feature and the enhanced output feature to obtain an input feature of the ith dynamic multi-scale convolution; extracting a plurality of angle features and respective weights of the plurality of angle features from the input feature of the ith dynamic multi-scale convolution; performing affine transformation on the coordinates of each pixel according to an angle feature closest to the coordinates of each pixel in the plurality of angle features to obtain transformed coordinates of each pixel; performing bilinear interpolation on the convolution kernel weight of each pixel based on the transformed coordinates of each pixel to obtain an interpolated convolution kernel weight of each pixel; performing weighted sum operation on the interpolated convolution kernel weight of each pixel and the weight of the angle feature closest to the coordinates of each pixel in the plurality of angle features to obtain a direction perception feature; and multiplying the direction perception feature and the input feature of the ith dynamic multi-scale convolution to obtain the ith rotation feature.

[0012] According to an embodiment of the present application, the preprocessing of the input image to obtain an input feature comprises: inputting the input image into a backbone network to obtain a first feature and a second feature of the input image; and inputting the first feature and the second feature into a third convolution network for feature fusion to obtain the input feature.

[0013] The second aspect of the present application provides an image processing device, comprising: a preprocessing module configured to preprocess an input image to obtain input features; a decoupling module configured to input the input features into a visual encoder and a multi-layer perception machine in a visual center decoupler to obtain enhanced specialized features and salient object detection specialized features; the visual encoder is configured to aggregate local region features based on the input features to obtain the enhanced specialized features, and the multi-layer perception machine is configured to capture edge information based on the input features to obtain the salient object detection specialized features; an enhancement module configured to input the enhanced specialized features into an enhancement network to obtain enhanced output features; wherein the enhancement network is configured to take illumination weights of different color channels and local binary pattern features of the input image as illumination constraints, and enhance the enhanced specialized features to obtain the enhanced output features; and a salient object detection module configured to input the salient object detection specialized features and the enhanced output features into a salient object detection network to detect salient objects.

[0014] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method.

[0015] The fourth aspect of the present application further provides a computer-readable storage medium having stored thereon a computer program or instructions, which, when executed by a processor, implement the steps of the method.

[0016] The fifth aspect of the present application further provides a computer program product comprising a computer program or instructions, which, when executed by a processor, implement the steps of the method.

[0017] According to the embodiments of the present application, the features learned from the input features are decoupled into enhanced specialized features and salient object detection specialized features by the visual center decoupler, which can alleviate the conflict between the enhancement task and the salient object detection task of the low-light image to a certain extent in the salient object detection task in a low-light environment. The introduction of illumination weights of different color channels and the aggregation of local binary pattern features of the input image for illumination constraints in the enhancement task of the low-light image can optimize the overall brightness and quality of the low-light image while improving the texture stability of the low-light image. In the salient object detection task of the low-light image, the effective integration of multi-scale required fine-grained details can ensure the clarity of details, retain the key edge information required for salient object detection, improve the overall detection efficiency and detection performance, and improve the salient object detection effect in a low-light scene. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the present application will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0019] Figure 1 An application scenario diagram of the image processing method according to an embodiment of the present application is shown.

[0020] Figure 2 An operation flow diagram of the image processing method according to an embodiment of the present application is shown.

[0021] Figure 3 A schematic diagram of the image processing method according to an embodiment of the present application is shown.

[0022] Figure 4 A schematic diagram of the visual center decoupler according to an embodiment of the present application is shown.

[0023] Figure 5 A schematic diagram of the dynamic multi-scale convolution according to an embodiment of the present application is shown.

[0024] Figure 6 A schematic diagram of the detection result of the salient object according to an embodiment of the present application is shown.

[0025] Figure 7 A schematic diagram of the image processing apparatus according to an embodiment of the present application is shown.

[0026] Figure 8 A block diagram of the electronic device suitable for implementing the image processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary of the present application, and is not intended to limit the scope of the present application. Throughout the specification, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that one or more embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring the concept of the present application.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so on, mean the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0029] All terms used herein, including technical and scientific terms, have the meaning commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the use of any of the terms used herein should not be interpreted in an idealized or overly formal sense unless specifically so stated.

[0030] In the case of using expressions such as "at least one of A, B, and C", etc., it should generally be interpreted that the meaning is the same as "A or B or C or any combination of A, B, and C" unless otherwise defined.

[0031] The salient object refers to the foreground region in the image that can spontaneously attract human visual attention. The salient object can have obvious differences in visual features such as color, brightness, texture, shape, etc. with the surrounding background. The saliency is a quantitative indicator for measuring the strength of the attraction of the image region to human visual attention.

[0032] Using other modalities such as thermal infrared images, depth images, etc. to assist salient object detection can improve the performance of salient object detection in complex scenes, but the effect is not good. For low-light images obtained in low-light conditions, the technical ideas for salient object detection include "first enhancing the low-light image, and then performing target detection", but enhancing the low-light image sometimes does not significantly improve the performance of salient object detection, and some enhancement methods may even have a negative transfer effect on the performance of salient object detection, and the influence of the same enhancement method on different salient object detection methods is different.

[0033] The salient object detection problem of low-light images can be solved by a neural network model of an end-to-end framework, but the enhancement task of low-light images and the salient object detection task require different features, resulting in conflicts in the training process of the neural network model. First, the enhancement task of low-light images usually focuses on the overall brightness and quality of the image, while the salient object detection task of low-light images pays more attention to local contrast and target edge accuracy. Global enhancement may cause local details to be ignored, and local optimization may affect the overall quality of the image. Second, noise suppression is a key step in the enhancement task of low-light images, but the noise suppression process may cause image details to be lost, and the salient object detection task of low-light images requires clear details and edge information, and excessive noise suppression will weaken the detection effect. Finally, the enhancement task of low-light images adjusts the image color and contrast to improve the overall visual effect, but it may change the features of the salient object, making it difficult for the salient object detection to accurately identify the target. Therefore, the inconsistency of the optimization goals of the enhancement task of low-light images and the salient object detection task limits the development of the salient object detection of low-light images.

[0034] Embodiments of the present application provide an image processing method, which comprises pre-processing an input image to obtain input features; inputting the input features into a visual center decoupler to decouple the features, to obtain enhanced special features and salient object detection special features; inputting the enhanced special features into an enhancement network to obtain enhanced output features; and inputting the salient object detection special features and the enhanced output features into a salient object detection network to detect salient objects.

[0035] Figure 1 An application scenario diagram of the image processing method according to an embodiment of the present application is shown.

[0036] As shown in Figure 1 , the application scenario 100 according to this embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0037] A user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0039] The server 105 can be a server providing various services, such as a background management server supporting a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data, etc. obtained or generated according to user requests) to the terminal device.

[0040] It should be noted that the image processing method provided in the embodiments of the present application can be generally executed by the server 105. Correspondingly, the image processing apparatus provided in the embodiments of the present application can be generally arranged in the server 105. The image processing method provided in the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the image processing apparatus provided in the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the image processing method provided in the embodiments of the present application can also be executed by the first terminal device 101, the second terminal device 102 and / or the third terminal device 103. Correspondingly, the image processing apparatus provided in the embodiments of the present application can also be arranged in the first terminal device 101, the second terminal device 102 and / or the third terminal device 103.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the above-mentioned system is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

[0042] Figure 2 An operation flowchart of the image processing method according to the embodiments of the present application is shown.

[0043] As Figure 2 shown, the image processing method of this embodiment includes operations S210-S240.

[0044] In operation S210, the input image is preprocessed to obtain input features.

[0045] In operation S220, the input features are respectively input into a visual encoder and a multi-layer perception machine in the visual center decoupler to obtain enhanced specialized features and salient object detection specialized features. The visual encoder is used to aggregate local region features based on the input features to obtain the enhanced specialized features, and the multi-layer perception machine is used to capture edge information based on the input features to obtain the salient object detection specialized features.

[0046] In operation S230, the enhanced specialized features are input into an enhancement network to obtain enhanced output features. The enhancement network is used to take the illumination weight light constraint of different color channels and the local binary pattern features of the input image as light constraints, and enhance the enhanced specialized features to obtain the enhanced output features.

[0047] In operation S240, the salient object detection specialized features and the enhanced output features are input into a salient object detection network to detect salient objects.

[0048] Figure 3 A schematic diagram of an image processing method according to an embodiment of the present application is shown.

[0049] As Figure 3 shown, the visual center decoupler includes a visual encoder and a multi-layer perception machine. The visual encoder is configured to aggregate local region features based on input features to obtain enhanced specialized features. The multi-layer perception machine is configured to capture edge information based on the input features to obtain salient object detection specialized features. An enhancement network is configured to enhance the enhanced specialized features based on illumination weights of different color channels and local binary pattern features of the input image as illumination constraints to obtain enhanced output features. A salient object detection network is configured to detect salient objects based on the salient object detection specialized features and the enhanced output features.

[0050] The input image can be a low-light image, which can be an image with low overall brightness, poor contrast, obvious noise, and dark colors, acquired under the condition that the ambient illuminance is less than or equal to 1 lux, or the average pixel value is less than or equal to 50, or the photosensitive sensitivity of the camera is greater than or equal to 3200. For low-quality degradation factors, such as quality loss caused by image acquisition, transmission, or storage, a special network needs to be designed to assist in salient object detection. In one example, as Figure 3 shown, the input image is a low-light image in a low-light environment, and the light of the street lamp in the low-light image is the most salient feature.

[0051] A convolutional neural network can be used to preprocess the low-light image. The convolutional neural network can include components such as convolutional layers and pooling layers, and can automatically learn and extract features such as edges, textures, and object shapes of the low-light image to obtain input features.

[0052] The enhancement task and the salient object detection task of the low-light image have different requirements, and will conflict due to the difference in the training process. The enhancement task of the low-light image can be understood as improving the visual quality and the amount of useful information of the low-light image, which is not simply brightness stretching. In the enhancement task of the low-light image, while improving the brightness or contrast of the low-light image, harmful information or artifacts introduced by the original imaging conditions or the enhancement algorithm itself will also be identified and suppressed, that is, while increasing visibility, degradation removal will also be performed to remove impurities. Therefore, enhancement and degradation removal need to be performed in coordination. However, degradation removal will weaken the clear edges, textures, and contrasts in the low-light image, resulting in loss of details and negative transfer effects on the salient object detection task of the low-light image, i.e., reducing the performance of salient object detection.

[0053] The visual encoder can obtain local region features of the input features and aggregate the local region features. The local region features can represent contrast, brightness, color, and the like of the input features, and can be used for the enhancement task of the low-light image. The multi-layer perception can capture edge information based on the input features, such as edge details of the foreground and the background of the input features, and can be used for the salient object detection task of the low-light image. The visual encoder and the multi-layer perception using the visual center decoupler learn features from the input features, and decouple the learned features into features required for two different tasks: enhancement-specific features for the enhancement task of the low-light image and salient object detection-specific features for the salient object detection task of the low-light image.

[0054] The introduction of the illumination constraint and the local binary pattern of the input image in the enhancement network can achieve illumination constraint and image texture stability maintenance, and reduce the negative transfer effect of degradation removal on salient object detection. Since the effects of illumination on different color channels are different, the illumination constraint is performed on each color channel of the enhancement-specific features, and the illumination constraint can be a three-primary-color (RGB) illumination constraint. Under real-world physical illumination, the changes of the RGB three color channels of each pixel point in the image are usually not completely independent and random, but have certain internal correlation and physical laws. The three-primary-color illumination constraint can guide the enhancement network to the direction consistent with the physical illumination change, to ensure the naturalness and authenticity of the result. The lighting weight can quantitatively control the coefficient of the contribution degree of the color channel to the overall illumination of the image. Illumination is a global feature, and has little effect on the pixel ordering of the local binary pattern (LBP) in a small image block, so that the local binary pattern is not sensitive to illumination changes. The local binary pattern of the input image represents a feature obtained by comparing the intensity of a pair of pixel points in a local region and encoding the comparison result into a binary string (0 or 1), which describes the relative gray scale relationship of the local region. When the illumination condition changes, the output of the local binary pattern feature remains stable, that is, it is not sensitive to illumination.

[0055] The salient object detection network can fuse the salient object detection-specific features and the enhancement output features, integrate multi-scale fine-grained details, obtain edge detail information required for the salient object detection task, and output the salient object detection result. As shown in FIG. 6, the light of the street lamp in the low-light image is taken as a salient object in the detection result. Figure 3

[0056] ​According to an embodiment of the present application, by using the visual center decoupler, the features learned from the input features are decoupled into the enhancement dedicated features and the salient object detection dedicated features, which can relieve the conflict between the enhancement task and the salient object detection task of the low-light image to a certain extent. In the enhancement task of the low-light image, the introduction of the illumination weight of different color channels and the aggregation of the local binary pattern features of the input image for illumination constraint can optimize the overall brightness and quality of the low-light image while improving the texture stability of the low-light image. In the salient object detection task of the low-light image, the effective integration of the multi-scale required fine-grained details can ensure the detail clarity, retain the key edge information required by the salient object detection, improve the overall detection efficiency and detection performance, and improve the salient object detection effect in the low-light scene.

[0057] According to an embodiment of the present application, the enhancement network and the salient object detection network can be combined, and a joint optimization training model can be used to balance the requirements of the enhancement task and the salient object detection task of the low-light image. For example, independent loss functions are defined for the enhancement network and the salient object detection network respectively, the loss values of the outputs of the enhancement network and the salient object detection network are calculated respectively, and the total loss function is optimized in a weighted manner to ensure that the dual-task requirements of the enhancement task and the salient object detection task of the low-light image are met.

[0058] According to an embodiment of the present application, as shown in Figure 3 The preprocessing of the input image can include: inputting the input image into a backbone network (Pyramid Vision Transformer, PVT) to obtain first features and second features of the input image; and inputting the first features and the second features into a third convolutional network for feature fusion to obtain input features. The first features and the second features represent feature maps output by the backbone network at different depths. The convolutional blocks in the first third of the backbone network can learn the texture, color combination and simple shape features of the input image, which are called shallow information, and output the first features. The convolutional blocks in the last third of the backbone network can learn the high-level semantics and global context of the input image, which are called deep information, and output the second features. The third convolutional network can include convolutional layers, batch normalization layers and activation layers. The third convolutional network can perform smoothing processing and feature fusion on the first features and the second features through convolution operation, batch normalization and activation operation to obtain the input features.

[0059] According to an embodiment of the present application, the input features are input into a visual encoder (Visual Encoding Mechanism, VEM) to obtain first output features . The input features inputting a multi-layer perceptron (MLP) to obtain a second output feature . The first output feature and the second output feature are spliced to obtain a third output feature . The third output feature is decoupled by using a first convolutional network to obtain an enhanced special feature . The third output feature is decoupled by using a second convolutional network to obtain a salient object detection special feature . The above process can be represented as:

[0060] (1);

[0061] (2);

[0062] (3);

[0063] wherein, represents splicing (concat). The first output feature and the second output feature are spliced and fused along the channel dimension of the input feature to obtain the third output feature . The third output feature is decoupled to perform linear projection in the fused feature space through the task-driven channel dimension, and the enhanced special feature and the salient object detection special feature are obtained, respectively. is a first convolutional network; is a second convolutional network. The first convolutional network can include a 3x3 size convolutional kernel and an activation layer, and the first convolutional network is a shallow convolutional network. The shallow convolutional network can extract shallow information in the third input feature, and the first output feature mainly includes shallow information. The second convolutional network can include multiple 3x3 size convolutional kernels, and each convolutional kernel is connected with an activation layer. The second convolutional network can perform multiple convolution operations in succession to form a deep convolutional network. The deep convolutional network can extract deep information in the third input feature, and the second output feature mainly includes deep information.

[0064] According to embodiments of the present application, the visual encoder can learn local region features such as contrast, brightness, and color of the input feature, and the multi-layer perceptron can capture global information of deep information.

[0065] Figure 4 A schematic diagram of a visual center decoupler according to an embodiment of this application is shown.

[0066] like Figure 4 As shown, the visual center decoupler includes two aggregation paths. The first aggregation path is the data processing path of the visual encoder. Figure 4 The right-hand path in the middle), the second aggregation path is the data processing path of the multilayer perceptron ( Figure 4 (The left path in the middle).

[0067] According to an embodiment of this application, the input features are mapped pixel by pixel to multiple pixel features to obtain a first input set; the first input set is encoded based on the visual center codebook to obtain the first position features corresponding to each of the multiple visual centers in the visual center codebook; the first position features corresponding to each of the multiple visual centers are fused to obtain a second position feature; the first input set and the second position feature are multiplied at the channel level to obtain a local region feature; the first input set and the local region feature are added at the channel level to obtain a first output feature.

[0068] like Figure 4 As shown, a visual encoder may include a convolutional neural network and a visual central codebook.

[0069] Input features can be processed through convolutional neural networks. The first input set is obtained by mapping each pixel of the C×H×W dimension to a set of C-dimensional feature vectors. : ,in, , Representing input features The total number of pixel features, It can represent input features The spatial location of the corresponding feature map in a two-dimensional plane. First input set. The pixel features It is a 1×C dimensional vector. The value is an integer. In one example, a convolutional neural network may consist of a convolutional layer, a batch normalization layer, a ReLU activation layer, another convolutional layer, another convolutional layer, a batch normalization layer, and a ReLU activation layer in sequence.

[0070] A visual encoder is a deep learning model that uses learnable codewords as visual centers. In one example, a visual center codebook can include K visual centers (codewords or keys) and a set of smoothing factors. Each visual center is a 1×C dimensional codeword vector, connected to the first input set. Each pixel feature has the same dimension. First input set Each pixel feature is matched with K codeword vectors, and a smoothing factor can be used to adjust the smoothness of codeword matching. The visual encoder can learn a visual central codebook and apply it to the first input set. Encode the codeword and output the value corresponding to each codeword vector, which is the first positional feature. K is a positive integer.

[0071] For the first input set The pixel features and the Each codeword vector Based on the first input set The pixel features and the Each codeword vector The difference , or in combination with the first Each codeword vector Smoothing factor The difference after that can be generated by the neural network layer. Learnable weights . No. Learnable weights It can be represented as:

[0072] (4);

[0073] in, This represents the L2 parameter operation, also known as the squared penalty. It is a natural constant. Indicates the first One smoothing factor; Indicates the first Each codeword vector.

[0074] The first Learnable weights Multiply by the difference The first input set can be obtained. The pixel features Compared to the first The position of each codeword (visual center). Then, the entire first input set... Compared to the first The position of the first code character (visual center), i.e., the first... The first location feature corresponding to each visual center It can be represented as:

[0075] (5);

[0076] fuse the first position features corresponding to each of the plurality of visual centers to obtain a second position feature . The second position feature may be represented as:

[0077] (6) ;

[0078] wherein, and are both 1xC-dimensional vectors. denotes a Batch Normalization (BN) layer with a ReLU activation function.

[0079] The second position feature is mapped to Cx1x1 dimensions using a fully connected layer to be used as an influence factor. The first input set is multiplied with the second position feature at a channel level to obtain a local region feature . The local region feature may be represented as:

[0080] (7) ;

[0081] wherein, denotes a fully connected layer; denotes a Sigmoid activation function; denotes a channel-level multiplication.

[0082] The first input set and the local region feature are added to obtain a first output feature . The first output feature may be represented as:

[0083] (8) ;

[0084] wherein, denotes a channel-level addition.

[0085] As shown in Figure 4 , the Batch Normalization layer, the ReLU activation layer, and the fully connected layer can be used for post-processing operations of the visual encoder. The first output feature is reshaped to obtain a first reshaped image, which can represent the first output feature mainly characterizes the feature information of the visual center.

[0086] ​According to embodiments of this application, a visual encoder can use a neural network to learn the center vectors (codeword vectors) of K visual centers end-to-end, and perform distance-weighted aggregation of the pixel features of the input features with the center vectors to obtain channel attention with global contextual semantics.

[0087] According to an embodiment of this application, the input features are grouped along the channel dimension to obtain a second input set; the second input set is input into a depth convolution unit to perform independent convolution operations on the features of each channel of the second input set, thereby capturing edge information and obtaining a convolved second input set; the convolved second input set is input into a channel multilayer perceptron unit to fuse the features of each channel of the convolved second input set to obtain a second output feature.

[0088] like Figure 4 As shown, a multilayer perceptron may include convolutional layers, depthwise convolutional units, and channel multilayer perceptron units.

[0089] Input features can be processed along the channel dimension (channel C dimension) using convolutional layers. Grouping can be done, for example, by uniformly dividing along channel C. Group, to obtain the second input set : Second input set The Feature map It is p×H×W dimension, where, . It is an integer.

[0090] In one example, such as Figure 4 As shown, a depthwise convolutional unit may include a group normalization layer, a depthwise convolutional layer, a channel scaling layer, and a path dropout layer. The group normalization layer and the depthwise convolutional layer are used to process the second input set. The data is processed, including channel scaling and path discarding operations, and then compared with the second input set. Perform residual connections to obtain the second input set after convolution. The second input set after convolution. It can be represented as:

[0091] (9);

[0092] in, Indicates group normalization; This indicates a depthwise convolution with a 1×1 kernel; Indicates channel scaling; This indicates that the path was discarded.

[0093] In one example, such as Figure 3As shown, the channel multi-layer perception unit can include a group normalization layer, a channel multi-layer perception layer, a channel scaling layer, and a path dropout layer. The group normalization layer and the channel multi-layer perception layer are used to process the second input set after convolution , and perform channel scaling and path dropout operations, and then perform residual connection with the second input set after convolution to obtain the second output feature . The second output feature may be represented as:

[0094] (10) ;

[0095] wherein, represents the channel multi-layer perception.

[0096] The second output feature is reshaped to obtain a second reshaped image, and the second reshaped image can represent the second output feature mainly representing the feature information of the boundary.

[0097] According to embodiments of the present application, the input feature is grouped along the channel dimension to obtain the second input set, which can improve the feature representation while reducing the calculation cost. The deep convolution unit and the channel multi-layer perception unit are used to process the second input set, which can effectively reduce the calculation complexity and effectively decode the visual feature. The channel scaling and path dropout operations can improve the model generalization and robustness.

[0098] In the enhancement network, according to embodiments of the present application, each color channel of the enhancement special feature is inverted; an illumination weight is applied to each inverted color channel and spliced to obtain an attention weight map; a texture feature of the input image is extracted based on a local binary pattern feature of the input image; and the attention weight map and the texture feature are spliced to obtain an enhancement output feature.

[0099] The gray scale can reflect the illumination distribution of the whole image, but the influence of illumination on different color channels of the enhancement special feature (image) is not the same. For example, the red color channel of the enhancement special feature is more obviously affected by illumination, and the light and dark contrast is stronger; the blue color channel of the enhancement special feature is less affected, and the light and dark difference is smaller.

[0100] The color inversion operation is performed on each color channel of the enhanced special feature respectively, which changes the bright area to dark and the dark area to bright, so that the illumination-related features of each color channel can be adjusted more accurately, and better processing effect can be achieved. The inversion operation means that the intensity of each pixel in a single color channel is mapped to the complementary value of its corresponding color gamut, so that the bright part becomes dark and the dark part becomes bright. Different color channels of the enhanced special feature are very sensitive to changes in illumination intensity. After the illumination constraint, the features of each color channel can no longer fluctuate sharply with changes in illumination conditions, which ensures the accuracy of subsequent classification, detection, matching and other tasks. For example, in face recognition, after the illumination constraint is performed on each color channel of the three primary colors of the face image, the extracted features can remain consistent regardless of whether the face is in strong light, weak light or side light environment, which can prevent misjudgment caused by different illumination.

[0101] The illumination weight of each color channel can be expressed as:

[0102] (11) ;

[0103] wherein, represents the illumination weight of the pixel of the enhanced special feature at position in a single color channel, represents the intensity of the pixel of the enhanced special feature after the inversion operation at position . and are the maximum value and the minimum value of the intensity of the pixel of the enhanced special feature after the inversion operation, respectively; and are the maximum value and the minimum value of the intensity of the pixel of the enhanced special feature after the inversion operation in the current color channel, respectively.

[0104] The illumination weight is applied to each color channel after inversion to perform illumination constraint, and the attention weight map is obtained after splicing.

[0105] (12) ;

[0106] wherein, represents the merging of the three single color channels; represents the illumination weight of the pixel of the enhanced special feature after the inversion operation at position in the red color channel; represents the illumination weight of the pixel of the enhanced special feature after the inversion operation at position in the green color channel; represents the illumination weight of the pixel of the enhanced special feature after the inversion operation at position in the blue color channel. represents the illumination weight of the pixel after merging the three single color channels after the reverse operation of the enhanced dedicated feature.

[0107] The texture feature of the input image is extracted based on the local binary pattern feature of the input image. The input image is divided into a plurality of 16x16 small regions. For the pixels in each small region, the gray values of the 8 neighborhood pixels are compared with the center point pixel. The gray value of each neighborhood pixel is marked as 1 if it is greater than the gray value of the center point pixel, and marked as 0 if it is less than or equal to the gray value of the center point pixel. An 8-bit binary number can be generated as the value of the center point pixel of each small region. The frequency of the value of each pixel in each small region is calculated, and the histogram is normalized. The histograms of each small region are spliced to form a feature vector, i.e. the texture feature (Illumination-Invariant Local, ILL) of the entire image of the input image.

[0108] The attention weight map and the texture feature are spliced to obtain an enhanced output feature.

[0109] According to an embodiment of the present application, by using the illumination constraint of the color channel, the overall brightness of the image can be improved while the details of the input image are preserved to prevent the negative impact of detail loss and image blur on the salient object detection task of the low-light image. Based on the local binary pattern of the input image, the texture feature of the input image is extracted, which can maintain the stability of the texture feature of the input image and improve the accuracy of edge detection in the salient object detection task of the low-light image.

[0110] As shown in Figure 5 , the salient object detection network can include G dynamic multi-scale convolutions (Dynamic Multi-Scale Convolution, DMC).

[0111] According to an embodiment of the present application, for the first dynamic multi-scale convolution, the salient object detection dedicated feature is input into the first dynamic multi-scale convolution to obtain the first rotation feature . For the i-th dynamic multi-scale convolution, the (i-1)-th rotation feature and the enhanced output feature are input into the i-th dynamic multi-scale convolution to obtain the i-th rotation feature. 1

[0112] According to an embodiment of the present application, for the i-th dynamic multi-scale convolution, the (i-1)-th rotation feature and the enhanced output feature are spliced to obtain the input feature of the i-th dynamic multi-scale convolution. The (i-1)-th rotation feature and the image size of the enhanced output feature is consistent with the input feature. The input feature of the i-th dynamic multi-scale convolution is consistent with the input feature.

[0113] Figure 5 A schematic diagram of the dynamic multi-scale convolution according to an embodiment of the present application is shown.

[0114] As shown in Figure 5 , in one example, the dynamic multi-scale convolution can include a dynamic block, a multi-weight rotation layer, a rotation weight layer, and a convolution layer.

[0115] A plurality of angle features and a plurality of respective weights of the angle features are extracted from the input feature of the i-th dynamic multi-scale convolution .

[0116] (13) ;

[0117] wherein, represents the dynamic block, and the convolution kernel weight can be re-parameterized according to the predicted angle feature. The dynamic block can be used to extract the direction feature and predict a plurality of angle features and respective weights . In one example, as shown in Figure 5 , the dynamic block can include a deep convolution layer, a layer normalization layer, a ReLU activation layer, an adaptive average pooling layer, and a linear layer.

[0118] According to the angle feature closest to the coordinate of each pixel among the plurality of angle features, the coordinate of each pixel is subjected to an affine transformation to obtain the transformed coordinate of each pixel.

[0119] (14) ;

[0120] wherein, represents an inverse matrix operation, represents the angle feature closest to the coordinate of the i-th pixel among the C angle features, represents the transformed coordinate of the i-th pixel by subjecting the coordinate of the i-th pixel to an affine transformation with the angle feature around the i-th pixel. .

[0121] Based on the transformed coordinate of each pixel, the convolution kernel weight of each pixel is subjected to a bilinear interpolation to obtain the interpolated convolution kernel weight of each pixel.​​​

[0122] (15);

[0123] in, This represents bilinear interpolation. For the ... Convolution kernel weights of 1 pixel By performing bilinear interpolation, we can obtain the first... Interpolated convolution kernel weights for each pixel . No. The coordinates of the nth pixel change after the affine transformation. coordinates of pixels and the The transformed coordinates of each pixel There is a certain distance between them, the size of the feature map increases, and the weights of the convolution kernel change.

[0124] The orientation-aware features are obtained by weighting the interpolated convolution kernel weights of each pixel and the weights of the angle features closest to the coordinates of each pixel among multiple angle features.

[0125] Combine orientation-aware features with the input features of the i-th dynamic multi-scale convolution. Multiply to obtain the i-th rotation feature. .

[0126] (16);

[0127] Indicates the first The rotation weight of the nth pixel, The rotation weights of each pixel are determined by Corresponding weights and the Interpolated convolution kernel weights for each pixel Multiply them to get the result.

[0128] In one example, such as Figure 6 As shown, a multi-weighted rotation layer can be used to adjust the angular features of each pixel. and the weight of each pixel A bilinear interpolation method is used to perform multi-weight rotation operations on all pixels, obtaining the interpolated convolutional kernel weights for each pixel. The rotation weight layer can be used to sum the rotation weights of each pixel to generate orientation-aware features. These orientation-aware features are then combined with the input features of the i-th dynamic multi-scale convolution. Multiply to obtain the i-th rotation feature. Convolutional layers can be used to process the i-th rotated feature. Perform a convolution operation to adjust the i-th rotated feature. scale information for subsequent operations.

[0129] In one example, the salient object detection network can include 3 dynamic multi-scale convolutions, and the input channel sizes of the 3 dynamic multi-scale convolutions are 320, 128, and 64 in sequence.

[0130] According to the embodiment of the present application, the i-1th rotation feature and the enhanced output feature are spliced, which can prevent the negative transfer effect caused by detail loss and image blur on the salient object detection task. In the salient object detection network, the angle feature is captured from the salient object detection dedicated feature, and the convolution kernel weight is dynamically re-parameterized according to the angle feature and the weight, which can filter the redundant features.

[0131] Figure 6 A schematic diagram of a salient object detection result according to an embodiment of the present application is shown.

[0132] As shown in Figure 7 , the first column is 5 low-light images, the second column is the true value map of the 5 low-light images, the third column is the salient object detection result obtained by processing the 5 low-light images respectively by using an ADMNet (Attention-guided Densely Multi-scale Network) image processing method. The fourth column is the salient object detection result obtained by processing the 5 low-light images respectively by using a MINet (Multi-scale Interactive Network) image processing method. The fifth column is the salient object detection result obtained by processing the 5 low-light images respectively by using a SI-SOD (Size-Invariant Salient Object Detection) image processing method. The sixth column is the salient object detection result obtained by processing the 5 low-light images respectively by using a UGRAN (Uncertainty-Guided Refinement Attention Network) image processing method. The seventh column is the salient object detection result obtained by processing the 5 low-light images respectively by using the image processing method of the present application. The salient object detection result obtained by the image processing method of the present application is more accurate.

[0133] Figure 7 A schematic diagram of an image processing device according to an embodiment of the present application is shown.

[0134] As shown in Figure 8As shown, the image processing apparatus 700 comprises a preprocessing module 710, a decoupling module 720, an enhancement module 730 and a salient object detection module 740.

[0135] The preprocessing module 710 is configured to pre-process an input image to obtain input features.

[0136] The decoupling module 720 is configured to input the input features into a visual encoder and a multi-layer perception machine in a visual center decoupler respectively to obtain enhancement-specific features and salient object detection-specific features; the visual encoder is configured to aggregate local region features based on the input features to obtain the enhancement-specific features, and the multi-layer perception machine is configured to capture edge information based on the input features to obtain the salient object detection-specific features.

[0137] The enhancement module 730 is configured to input the enhancement-specific features into an enhancement network to obtain enhancement output features. The enhancement network is configured to take illumination weights of different color channels and local binary pattern features of the input image as illumination constraints, and enhance the enhancement-specific features to obtain the enhancement output features.

[0138] The salient object detection module 740 is configured to input the salient object detection-specific features and the enhancement output features into a salient object detection network to detect a salient object.

[0139] According to an embodiment of the present application, the decoupling module 720 comprises a visual encoder submodule, a multi-layer perception machine submodule, a splicing submodule, a first decoupling submodule and a second decoupling submodule.

[0140] The visual encoder submodule is configured to input the input features into the visual encoder to obtain first output features.

[0141] The multi-layer perception machine submodule is configured to input the input features into the multi-layer perception machine to obtain second output features.

[0142] The splicing submodule is configured to splice the first output features and the second output features to obtain third output features.

[0143] The first decoupling submodule is configured to decouple the third output features by using a first convolutional network to obtain the enhancement-specific features.

[0144] The second decoupling submodule is configured to decouple the third output features by using a second convolutional network to obtain the salient object detection-specific features.

[0145] According to an embodiment of the present application, the visual encoder submodule comprises a mapping unit, a codebook encoding unit, a fusion unit, a multiplication unit and an addition unit.

[0146] The mapping unit is configured to map the input features into a plurality of pixel features pixel by pixel to obtain a first input set.

[0147] The codebook encoding unit is used to encode the first input set based on the visual center codebook to obtain the first position features corresponding to each of the multiple visual centers in the visual center codebook.

[0148] The fusion unit is used to fuse the first position features corresponding to multiple visual centers to obtain the second position features.

[0149] The multiplication unit is used to perform channel-level multiplication of the first input set and the second location feature to obtain local region features.

[0150] The addition unit is used to add the first input set and the local region features at the channel level to obtain the first output feature.

[0151] According to an embodiment of this application, the multilayer sensor submodule includes a grouping unit, a first convolutional unit, and a second convolutional unit.

[0152] The grouping unit is used to group the input features along the channel dimension to obtain the second input set.

[0153] The first convolutional unit is used to input the second input set into the depthwise convolutional unit to perform independent convolution operations on the features of each channel of the second input set, so as to obtain the convolved second input set.

[0154] The second convolutional unit is used to input the convolved second input set into the channel multilayer perceptron unit to fuse the features of each channel of the convolved second input set to obtain the second output feature.

[0155] According to an embodiment of this application, the enhancement module 730 includes a color inversion submodule, an illumination constraint submodule, a texture feature extraction submodule, and a feature splicing submodule.

[0156] The Color Inversion submodule is used to invert each color channel of the enhancement-specific feature.

[0157] The illumination constraint submodule is used to apply illumination weights to each inverted color channel and stitch them together to obtain an attention weight map.

[0158] The texture feature extraction submodule is used to extract texture features from the input image based on the local binary pattern features of the input image.

[0159] The feature concatenation submodule is used to concatenate the attention weight map and texture features to obtain enhanced output features.

[0160] According to embodiments of this application, the salient object detection network includes G dynamic multi-scale convolutions. The salient object detection module 740 includes a first rotation feature acquisition submodule and a second rotation feature acquisition submodule.

[0161] The first rotation feature acquisition submodule is used to input the salient object detection-specific features into the first dynamic multi-scale convolution to obtain the first rotation feature.

[0162] The second rotation feature acquisition submodule is used to input the (i-1)th rotation feature and the enhanced output feature into the i-th dynamic multi-scale convolution to obtain the i-th rotation feature.

[0163] An integer 1 < i ≤ G, where the Gth rotation feature is the output of the salient object detection network.

[0164] According to an embodiment of this application, the second rotation feature obtaining submodule includes a splicing unit, an extraction unit, a transformation unit, an interpolation unit, a weighted sum unit, and an obtaining unit.

[0165] The concatenation unit is used to concatenate the (i-1)th rotated feature and the enhanced output feature to obtain the input feature of the i-th dynamic multi-scale convolution.

[0166] The extraction unit is used to extract multiple angular features and their respective weights from the input features of the i-th dynamic multi-scale convolution.

[0167] The transformation unit is used to perform an affine transformation on the coordinates of each pixel based on the angular feature closest to the coordinates of each pixel among multiple angular features, so as to obtain the transformed coordinates of each pixel.

[0168] The interpolation unit is used to perform bilinear interpolation on the convolution kernel weights of each pixel based on the transformed coordinates of each pixel, so as to obtain the interpolated convolution kernel weights of each pixel.

[0169] The weighted sum unit is used to perform a weighted sum operation on the interpolated convolution kernel weights of each pixel and the weights of the angle features closest to the coordinates of each pixel among multiple angle features, to obtain the orientation-aware features.

[0170] The obtained unit is used to multiply the orientation-aware feature and the input feature of the i-th dynamic multi-scale convolution to obtain the i-th rotation feature.

[0171] According to an embodiment of this application, the preprocessing module 710 includes a first obtaining submodule and a second obtaining submodule.

[0172] The first submodule is used to input the input image into the backbone network and obtain the first and second features of the input image.

[0173] The second submodule is used to input the first and second features into the third convolutional network for feature fusion to obtain the input features.

[0174] According to embodiments of this application, any multiple modules among the preprocessing module 710, decoupling module 720, enhancement module 730, and salient target detection module 740 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the preprocessing module 710, decoupling module 720, enhancement module 730, and salient target detection module 740 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the preprocessing module 710, decoupling module 720, enhancement module 730, and salient target detection module 740 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0175] Figure 8 A block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of this application is shown.

[0176] like ​ As shown, an electronic device 800 according to an embodiment of this application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0177] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0178] According to embodiments of this application, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0179] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0180] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0181] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the image processing method provided in the embodiments of this application.

[0182] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0183] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0184] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0185] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0186] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0187] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0188] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. An image processing method, characterized in that, The image processing method includes: The input image is preprocessed to obtain input features; The input features are respectively input into the visual encoder and the multilayer perceptron in the visual center decoupler to obtain enhancement-specific features and salient target detection-specific features; the visual encoder is used to aggregate local region features based on the input features to obtain the enhancement-specific features, and the multilayer perceptron is used to capture edge information based on the input features to obtain the salient target detection-specific features; The enhancement-specific features are input into the enhancement network to obtain the enhancement output features; wherein, the enhancement network is used to enhance the enhancement-specific features by taking the illumination weights of different color channels and the local binary mode features of the input image as illumination constraints, so as to obtain the enhancement output features; The salient target detection-specific features and the enhanced output features are input into the salient target detection network to detect salient targets.

2. The image processing method according to claim 1, characterized in that, The step of inputting the input features into the visual encoder and multilayer perceptron in the visual center decoupler to obtain enhancement-specific features and salient object detection-specific features includes: The input features are input into the visual encoder to obtain the first output features; The input features are input into the multilayer perceptron to obtain the second output features; The first output feature and the second output feature are concatenated to obtain the third output feature; The third output feature is decoupled using the first convolutional network to obtain the enhanced special feature; The third output feature is decoupled using a second convolutional network to obtain the salient target detection-specific feature.

3. The image processing method according to claim 2, characterized in that, The step of inputting the input features into the visual encoder to obtain the first output features includes: The input features are mapped pixel by pixel to obtain a first input set; The first input set is encoded based on the visual center codebook to obtain the first position features corresponding to each of the multiple visual centers in the visual center codebook. The first position features corresponding to each of the multiple visual centers are fused to obtain the second position features; The first input set is multiplied with the second location feature at the channel level to obtain the local region feature; The first input set is added to the local region features at the channel level to obtain the first output feature.

4. The image processing method according to claim 2, characterized in that, The step of inputting the input features into the multilayer perceptron to obtain the second output features includes: The input features are grouped along the channel dimension to obtain the second input set; The second input set is input into a depthwise convolutional unit to perform independent convolution operations on the features of each channel of the second input set, resulting in a convolutional second input set. The convolutional second input set is input into the channel multilayer perceptron unit to fuse the features of each channel of the convolutional second input set to obtain the second output feature.

5. The image processing method according to claim 1, characterized in that, The step of inputting the enhancement-specific features into the enhancement network to obtain the enhancement output features includes: Invert each color channel of the enhanced special feature; Illumination weights are applied to each of the inverted color channels and then stitched together to obtain an attention weight map; Based on the local binary pattern features of the input image, extract the texture features of the input image; The attention weight map and the texture features are concatenated to obtain the enhanced output features.

6. The image processing method according to claim 1, characterized in that, The salient object detection network includes G dynamic multi-scale convolutions; the step of inputting the salient object detection-specific features and the enhanced output features into the salient object detection network to detect salient objects includes: For the first dynamic multi-scale convolution, the salient target detection-specific feature is input into the first dynamic multi-scale convolution to obtain the first rotation feature; For the i-th dynamic multi-scale convolution, the (i-1)-th rotation feature and the enhanced output feature are input into the i-th dynamic multi-scale convolution to obtain the i-th rotation feature; Where 1 < i ≤ G is an integer, and the Gth rotation feature is the output of the salient target detection network.

7. The image processing method according to claim 6, characterized in that, The step of inputting the (i-1)th rotation feature and the enhanced output feature into the i-th dynamic multi-scale convolution to obtain the i-th rotation feature includes: The (i-1)th rotation feature and the enhanced output feature are concatenated to obtain the input feature of the i-th dynamic multi-scale convolution; Extract multiple angular features and their respective weights from the input features of the i-th dynamic multi-scale convolution; Based on the angle feature closest to the coordinates of each pixel among the multiple angle features, perform an affine transformation on the coordinates of each pixel to obtain the transformed coordinates of each pixel; Based on the transformed coordinates of each pixel, bilinear interpolation is performed on the convolution kernel weights of each pixel to obtain the interpolated convolution kernel weights of each pixel. The interpolated convolution kernel weights of each pixel and the weights of the angle features closest to the coordinates of each pixel among the multiple angle features are weighted and summed to obtain the orientation-aware features. The i-th rotation feature is obtained by multiplying the orientation-aware feature with the input feature of the i-th dynamic multi-scale convolution.

8. The image processing method according to claim 1, characterized in that, The preprocessing of the input image to obtain input features includes: The input image is input into the backbone network to obtain the first and second features of the input image; The first feature and the second feature are input into a third convolutional network for feature fusion to obtain the input feature.

9. An image processing apparatus, characterized in that, The image processing device includes: The preprocessing module is used to preprocess the input image to obtain input features; The decoupling module is used to input the input features into the visual encoder and multilayer perceptron in the visual center decoupling unit respectively to obtain enhancement-specific features and salient target detection-specific features; the visual encoder is used to aggregate local region features based on the input features to obtain the enhancement-specific features, and the multilayer perceptron is used to capture edge information based on the input features to obtain the salient target detection-specific features; An enhancement module is used to input the enhancement-specific features into an enhancement network to obtain enhanced output features; wherein, the enhancement network is used to enhance the enhancement-specific features by using the illumination weights of different color channels and the local binary mode features of the input image as illumination constraints to obtain the enhanced output features; and A salient target detection module is used to input the salient target detection-specific features and the enhanced output features into a salient target detection network to detect salient targets.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the image processing method according to any one of claims 1 to 8.

Images