Image processing method, apparatus and device

Abstract

The present application provides an image processing method, device and equipment, which can be applied to the technical field of image processing. The image processing method comprises: pre-processing an input image to obtain input features; inputting the input features into a visual encoder and a multi-layer perception machine in a visual center decoupler respectively to obtain enhanced special features and salient object detection special features; the visual encoder aggregates local region features based on the input features to obtain the enhanced special features, and the multi-layer perception machine captures edge information based on the input features to obtain the salient object detection special features; inputting the enhanced special features into an enhancement network to obtain enhanced output features; the enhancement network takes illumination weights of different color channels and local binary pattern features of the input image as illumination constraints, and enhances the enhanced special features to obtain the enhanced output features; and inputting the salient object detection special features and the enhanced output features into a salient object detection network to detect a salient object.

Description

Image processing methods, apparatus and equipment Technical Field

[0001] At least one embodiment of this application relates to the field of image segmentation technology, and more specifically to an image processing method, apparatus, and device. Background Technology

[0002] The core objective of salient object detection (SOD) is to identify and locate important and prominent regions in an image, helping computer vision systems quickly locate and identify key targets in complex scenes, thereby achieving more efficient image understanding and analysis. However, images acquired in low-light environments generally suffer from low contrast and poor distinction between targets and backgrounds, limiting the development of salient object detection technology.

[0003] Salience detection models primarily employ techniques such as pooling, downsampling, multi-scale analysis, and integrity learning to achieve more effective feature fusion, thereby improving model performance. However, low-light scenes are prevalent and unavoidable in daily life. When processing low-light data, most salient object detection models only perform simple fusion of foreground and background features, failing to effectively mitigate the interference of darkness degradation factors on foreground-background segmentation, resulting in low salient object detection performance. Summary of the Invention

[0004] In view of the above problems, this application provides an image processing method, apparatus and device for improving the performance of salient target detection.

[0005] According to a first aspect of this application, an image processing method is provided, comprising: preprocessing an input image to obtain input features; inputting the input features into a visual encoder and a multilayer perceptron in a visual center decoupler to obtain enhancement-specific features and salient target detection-specific features; the visual encoder aggregating local region features based on the input features to obtain the enhancement-specific features, and the multilayer perceptron capturing edge information based on the input features to obtain the salient target detection-specific features; inputting the enhancement-specific features into an enhancement network to obtain enhanced output features; wherein the enhancement network uses illumination weights of different color channels and local binary pattern features of the input image as illumination constraints to enhance the enhancement-specific features to obtain the enhanced output features; and inputting the salient target detection-specific features and the enhanced output features into a salient target detection network to detect salient targets.

[0006] According to an embodiment of this application, the above-mentioned input features are respectively input into the visual encoder and the multilayer perceptron in the visual center decoupler to obtain enhancement-specific features and salient target detection-specific features, including: inputting the input features into the visual encoder to obtain a first output feature; inputting the input features into the multilayer perceptron to obtain a second output feature; concatenating the first output feature and the second output feature to obtain a third output feature; decoupling the third output feature using a first convolutional network to obtain the enhancement-specific feature; and decoupling the third output feature using a second convolutional network to obtain the salient target detection-specific feature.

[0007] According to an embodiment of this application, the above-mentioned input features into the visual encoder to obtain a first output feature includes: mapping the input features pixel by pixel into multiple pixel features to obtain a first input set; encoding the first input set based on a visual center codebook to obtain first position features corresponding to each of the multiple visual centers in the visual center codebook; fusing the first position features corresponding to each of the multiple visual centers to obtain a second position feature; performing channel-level multiplication of the first input set and the second position feature to obtain the local region feature; and performing channel-level addition of the first input set and the local region feature to obtain the first output feature.

[0008] According to an embodiment of this application, the above-mentioned input features are input into the multilayer perceptron to obtain the second output features, which includes: grouping the input features along the channel dimension to obtain a second input set; inputting the second input set into a depthwise convolutional unit to perform independent convolution operations on the features of each channel of the second input set to obtain a convolutional second input set; and inputting the convolutional second input set into a channel multilayer perceptron unit to fuse the features of each channel of the convolutional second input set to obtain the second output features.

[0009] According to an embodiment of this application, the above-mentioned input of the enhancement-specific features into the enhancement network to obtain the enhancement output features includes: inverting each color channel of the above-mentioned enhancement-specific features; applying illumination weights to each inverted color channel and concatenating them to obtain an attention weight map; extracting the texture features of the above-mentioned input image based on the local binary pattern features of the above-mentioned input image; and concatenating the above-mentioned attention weight map and the above-mentioned texture features to obtain the above-mentioned enhancement output features.

[0010] According to an embodiment of this application, the aforementioned salient target detection network includes G dynamic multi-scale convolutions; the input of the aforementioned salient target detection-specific features and the aforementioned enhanced output features into the salient target detection network to detect salient targets includes: for the first aforementioned dynamic multi-scale convolution, inputting the aforementioned salient target detection-specific features into the first aforementioned dynamic multi-scale convolution to obtain a first rotation feature; for the i-th aforementioned dynamic multi-scale convolution, inputting the (i-1)-th rotation feature and the aforementioned enhanced output features into the i-th dynamic multi-scale convolution to obtain the i-th rotation feature; wherein, 1 < i ≤ G integers, and the G-th rotation feature is the output of the aforementioned salient target detection network.

[0011] According to an embodiment of this application, the above-mentioned inputting the (i-1)th rotation feature and the above-mentioned enhanced output feature into the i-th dynamic multi-scale convolution to obtain the i-th rotation feature includes: concatenating the (i-1)th rotation feature and the above-mentioned enhanced output feature to obtain the input feature of the i-th dynamic multi-scale convolution; extracting multiple angle features and their respective weights from the input feature of the i-th dynamic multi-scale convolution; performing an affine transformation on the coordinates of each pixel based on the angle feature closest to the coordinates of each pixel among the multiple angle features to obtain the transformed coordinates of each pixel; performing bilinear interpolation on the convolution kernel weights of each pixel based on the transformed coordinates of each pixel to obtain the interpolated convolution kernel weights of each pixel; performing a weighted sum operation on the interpolated convolution kernel weights of each pixel and the weight of the angle feature closest to the coordinates of each pixel among the multiple angle features to obtain the orientation-aware feature; and multiplying the orientation-aware feature with the input feature of the i-th dynamic multi-scale convolution to obtain the i-th rotation feature.

[0012] According to an embodiment of this application, the above-mentioned preprocessing of the input image to obtain input features includes: inputting the input image into a backbone network to obtain a first feature and a second feature of the input image; and inputting the first feature and the second feature into a third convolutional network for feature fusion to obtain the input features.

[0013] A second aspect of this application provides an image processing apparatus, comprising: a preprocessing module for preprocessing an input image to obtain input features; a decoupling module for inputting the input features into a visual encoder and a multilayer perceptron in a visual center decoupler to obtain enhancement-specific features and salient target detection-specific features; the visual encoder for aggregating local region features based on the input features to obtain the enhancement-specific features, and the multilayer perceptron for capturing edge information based on the input features to obtain the salient target detection-specific features; an enhancement module for inputting the enhancement-specific features into an enhancement network to obtain enhanced output features; wherein the enhancement network uses illumination weights of different color channels and local binary pattern features of the input image as illumination constraints to enhance the enhancement-specific features to obtain the enhanced output features; and a salient target detection module for inputting the salient target detection-specific features and the enhanced output features into a salient target detection network to detect salient targets.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] According to embodiments of this application, a visual center decoupler is used to decouple the features learned from the input features into enhancement-specific features and salient object detection-specific features. In the salient object detection task in low-light environments, this can alleviate the conflict between the low-light image enhancement task and the salient object detection task to some extent. In the low-light image enhancement task, introducing illumination weights for different color channels and aggregating local binary pattern features of the input image for illumination constraints can improve the overall brightness and quality of the low-light image while enhancing its texture stability. In the low-light image salient object detection task, effectively integrating the fine-grained details required at multiple scales ensures detail clarity, preserves the key edge information required for salient object detection, improves overall detection efficiency and performance, and enhances the salient object detection effect in low-light scenes. Attached Figure Description

[0018] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0019] Figure 1 illustrates an application scenario of the image processing method according to an embodiment of this application.

[0020] Figure 2 shows an operation flowchart of the image processing method according to an embodiment of this application.

[0021] Figure 3 shows a schematic diagram of an image processing method according to an embodiment of this application.

[0022] Figure 4 shows a schematic diagram of a visual center decoupler according to an embodiment of this application.

[0023] Figure 5 shows a schematic diagram of dynamic multi-scale convolution according to an embodiment of this application.

[0024] Figure 6 shows a schematic diagram of the detection results of a salient target according to an embodiment of this application.

[0025] Figure 7 shows a schematic diagram of an image processing apparatus according to an embodiment of this application.

[0026] Figure 8 shows a block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0031] A salient target is a foreground region in an image that spontaneously attracts human visual attention. A salient target can differ significantly from its surrounding background in visual features such as color, brightness, texture, and shape. Salientity is a quantitative measure of the strength of an image region's attractiveness to human visual attention.

[0032] Utilizing other modalities such as thermal infrared images and depth images to assist salient target detection can improve the performance of salient target detection in complex scenes, but the results are not ideal. For low-light images obtained under low-light conditions, the technical approach for salient target detection includes "enhancing the low-light image first, then performing target detection." However, enhancing the low-light image sometimes does not significantly improve the performance of salient target detection, and some enhancement methods may even have a negative transfer effect on salient target detection performance. Furthermore, the impact of the same enhancement method on different salient target detection methods varies.

[0033] While end-to-end neural network models can address the problem of salient object detection in low-light images, the different features required for low-light image enhancement and salient object detection lead to conflicts during neural network model training. First, low-light image enhancement typically focuses on overall image brightness and quality, while salient object detection prioritizes local contrast and target edge precision. Global enhancement may result in neglecting local details, while local optimization can negatively impact overall image quality. Second, noise suppression is a crucial step in low-light image enhancement, but this process can lead to loss of image details. Salient object detection in low-light images requires clear details and edge information; excessive noise suppression weakens detection performance. Finally, while low-light image enhancement improves overall visual quality by adjusting image color and contrast, it may alter the characteristics of salient objects, making accurate target identification difficult. Therefore, the inconsistency between the optimization objectives of low-light image enhancement and salient object detection limits the development of salient object detection in low-light images.

[0034] Embodiments of this application provide an image processing method, which includes preprocessing an input image to obtain input features; inputting the input features into a visual center decoupler to decouple the features, obtaining enhancement-specific features and salient target detection-specific features; inputting the enhancement-specific features into an enhancement network to obtain enhanced output features; and inputting the salient target detection-specific features and enhanced output features into a salient target detection network to detect salient targets.

[0035] Figure 1 illustrates an application scenario of the image processing method according to an embodiment of this application.

[0036] As shown in Figure 1, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0037] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0040] It should be noted that the image processing method provided in this application embodiment can generally be executed by server 105. Correspondingly, the image processing apparatus provided in this application embodiment can generally be located in server 105. The image processing method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the image processing apparatus provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the image processing method provided in this application embodiment can also be executed by the first terminal device 101, the second terminal device 102, and / or the third terminal device 103. Correspondingly, the image processing apparatus provided in this application embodiment can also be located in the first terminal device 101, the second terminal device 102, and / or the third terminal device 103.

[0041] It should be understood that the number of terminal devices, networks, and servers shown in Figure 1 is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] Figure 2 shows an operation flowchart of the image processing method according to an embodiment of this application.

[0043] As shown in Figure 2, the image processing method of this embodiment includes operations S210 to S240.

[0044] In operation S210, the input image is preprocessed to obtain input features.

[0045] In operation S220, the input features are fed into the visual encoder and multilayer perceptron in the visual center decoupler, respectively, to obtain enhancement-specific features and salient object detection-specific features. The visual encoder is used to aggregate local region features based on the input features to obtain enhancement-specific features, and the multilayer perceptron is used to capture edge information based on the input features to obtain salient object detection-specific features.

[0046] In operation S230, the enhancement-specific features are input into the enhancement network to obtain the enhanced output features. The enhancement network uses the illumination weights of different color channels and the local binary pattern features of the input image as illumination constraints to enhance the enhancement-specific features, thus obtaining the enhanced output features.

[0047] In operation S240, salient target detection-specific features and enhanced output features are input into the salient target detection network to detect salient targets.

[0048] Figure 3 shows a schematic diagram of an image processing method according to an embodiment of this application.

[0049] As shown in Figure 3, the visual center decoupler includes a visual encoder and a multilayer perceptron. The visual encoder aggregates local region features based on input features to obtain enhancement-specific features. The multilayer perceptron captures edge information based on input features to obtain salient object detection-specific features. The enhancement network uses illumination weights of different color channels and local binary pattern features of the input image as illumination constraints to enhance the enhancement-specific features, resulting in enhanced output features. The salient object detection network detects salient objects based on the salient object detection-specific features and the enhanced output features.

[0050] The input image can be a low-light image. A low-light image is defined as an image acquired under conditions where the ambient illumination is ≤1 lux, the average pixel value is ≤50, or the camera's light sensitivity is ≥3200, resulting in an overall image with low brightness, poor contrast, significant noise, and dull colors. To address quality degradation factors, such as quality loss during image acquisition, transmission, or storage, a dedicated network needs to be designed to assist in salient object detection. In one example, as shown in Figure 3, the input image is a low-light image taken in a low-light environment, where the streetlights are the most salient feature.

[0051] Convolutional neural networks (CNNs) can be used to preprocess low-light images. A CNN can include components such as convolutional layers and pooling layers, and can automatically learn and extract features such as edges, textures, and object shapes from low-light images to obtain input features.

[0052] The requirements of low-light image enhancement and salient object detection differ, leading to conflicts during training due to these differences. Low-light image enhancement can be understood as improving the visual quality and usable information of low-light images, not simply brightening. In low-light image enhancement, while increasing brightness or contrast, it also identifies and suppresses harmful information or artifacts introduced by poor original imaging conditions or the enhancement algorithm itself. That is, while increasing visibility, it also performs degradation removal to remove impurities. Therefore, enhancement and degradation removal need to be performed synergistically. However, degradation removal weakens originally sharp edges, textures, and contrast in low-light images, resulting in detail loss and a negative transfer effect on salient object detection in low-light images, i.e., reducing salient object detection performance.

[0053] A visual encoder can acquire and aggregate local region features from input features. These local region features can represent characteristics such as contrast, brightness, and color of the input features, and can be used for low-light image enhancement tasks. A multilayer perceptron can capture edge information based on input features, such as edge details between the foreground and background of the input features, and can be used for salient object detection tasks in low-light images. A visual encoder and a multilayer perceptron utilizing a visual center decoupler learn features from the input features, decoupling the learned features into two features required for different tasks: enhancement-specific features for low-light image enhancement and salient object detection-specific features for salient object detection in low-light images.

[0054] Introducing illumination constraints and local binary patterns (LBPs) of the input image into the augmentation network can maintain image texture stability while minimizing the negative transfer effect of degradation removal on salient object detection. Since illumination affects different color channels differently, illumination constraints are applied to each color channel of the augmentation-specific features. These constraints can be based on the three primary colors (RGB). Under real-world physical lighting, the changes in the RGB channels of each pixel in an image are usually not completely independent or random, but rather exhibit certain intrinsic correlations and physical laws. Primary color illumination constraints can guide the augmentation network in a direction consistent with physical lighting changes, ensuring the naturalness and realism of the results. Illumination weights can quantify the contribution of color channels to the overall image illumination. Illumination is a global feature and has little impact on the pixel ranking of the Local Binary Pattern (LBP) within small image patches, making the LBP insensitive to illumination changes. The local binary mode representation of the input image is a feature that compares the intensity of pixel pairs within a local region and encodes the comparison result as a binary string (0 or 1). It describes the relative grayscale relationship of the local region. The output of the local binary mode feature remains stable when the lighting conditions change, that is, it is not sensitive to lighting.

[0055] The salient object detection network can fuse salient object detection-specific features and enhanced output features, integrating multi-scale fine-grained details to obtain the edge detail information required for the salient object detection task, and outputting the salient object detection result. As shown in Figure 3, the detection result uses the light from streetlights in the low-light image as the salient object.

[0056] According to embodiments of this application, a visual center decoupler is used to decouple the features learned from the input features into enhancement-specific features and salient object detection-specific features. In the salient object detection task in low-light environments, this can alleviate the conflict between the low-light image enhancement task and the salient object detection task to some extent. In the low-light image enhancement task, introducing illumination weights for different color channels and aggregating local binary pattern features of the input image for illumination constraints can improve the overall brightness and quality of the low-light image while enhancing its texture stability. In the low-light image salient object detection task, effectively integrating the fine-grained details required at multiple scales ensures detail clarity, preserves the key edge information required for salient object detection, improves overall detection efficiency and performance, and enhances the salient object detection effect in low-light scenes.

[0057] According to embodiments of this application, an enhancement network and a salient object detection network can be combined, and a joint optimization training model can be used to balance the requirements of low-light image enhancement and salient object detection tasks. For example, independent loss functions are defined for the enhancement network and the salient object detection network respectively, the loss values ​​of the outputs of the enhancement network and the salient object detection network are calculated separately, and the total loss function is optimized in a weighted manner to ensure that the dual task requirements of low-light image enhancement and salient object detection are met.

[0058] According to an embodiment of this application, as shown in Figure 3, preprocessing the input image may include: inputting the input image into a Pyramid Vision Transformer (PVT) to obtain a first feature and a second feature of the input image; and inputting the first feature and the second feature into a third convolutional network for feature fusion to obtain the input features. The first feature and the second feature represent feature maps output by the Pyramid Vision Transformer at different depths. The first 1 / 3 or so of the convolutional blocks of the Pyramid Vision Transformer can learn the texture, color combinations, and simple shape features of the input image, referred to as shallow information, and output the first feature. The last 1 / 3 or so of the convolutional blocks of the Pyramid Vision Transformer can learn the high-level semantics and global context of the input image, referred to as deep information, and output the second feature. The third convolutional network may include convolutional layers, batch normalization layers, and activation layers. The third convolutional network can smooth and fuse the first feature and the second feature through convolution operations, batch normalization, and activation operations to obtain the input features.

[0059] According to embodiments of this application, the input features are... Inputting into a visual encoder (VEM) yields the first output feature. Input features Inputting into a multi-layer perceptron (MLP), the second output feature is obtained. The first output feature Second output features By concatenating the features, the third output feature is obtained. The third output feature is processed using the first convolutional network. Decoupling is performed to obtain enhanced specific features. The third output feature is processed using the second convolutional network. Decoupling is performed to obtain features specific to salient target detection. The above process can be represented as:

[0060] (1);

[0061] (2);

[0062] (3);

[0063] in, This represents concatenation. Along the input features... The channel dimension will be the first output feature Second output features The features are then spliced ​​and merged to obtain the third output feature. The third output feature Feature decoupling is performed, and in the fused feature space, linear projection is performed through the task-driven channel dimension to obtain the enhanced special features. Specialized features for salient target detection . This is the first convolutional network; The second convolutional network. The first convolutional network may include a 3×3 convolutional kernel and activation layers. The first convolutional network is a shallow convolutional network, which can extract shallow information from the third input features and the first output features. This primarily includes shallow information. The second convolutional network can include multiple 3×3 convolutional kernels, each followed by an activation layer. The second convolutional network can perform multiple convolutional operations consecutively to form a deep convolutional network. This deep convolutional network can extract deeper information from the third input features and the second output features. It mainly includes deep information.

[0064] According to embodiments of this application, a visual encoder can learn local region features such as contrast, brightness, and color of input features, while a multilayer perceptron can capture global information of deep information.

[0065] Figure 4 shows a schematic diagram of a visual center decoupler according to an embodiment of this application.

[0066] As shown in Figure 4, the visual center decoupler includes two aggregation paths. The first aggregation path is the data processing path of the visual encoder (the right path in Figure 4), and the second aggregation path is the data processing path of the multilayer perceptron (the left path in Figure 4).

[0067] According to an embodiment of this application, the input features are mapped pixel by pixel to multiple pixel features to obtain a first input set; the first input set is encoded based on the visual center codebook to obtain the first position features corresponding to each of the multiple visual centers in the visual center codebook; the first position features corresponding to each of the multiple visual centers are fused to obtain a second position feature; the first input set and the second position feature are multiplied at the channel level to obtain a local region feature; the first input set and the local region feature are added at the channel level to obtain a first output feature.

[0068] As shown in Figure 4, a visual encoder may include a convolutional neural network and a visual central codebook.

[0069] Input features can be processed through convolutional neural networks. The first input set is obtained by mapping pixel-by-pixel from C×H×W dimensions to a set of C-dimensional feature vectors. : ,in, , Representing input features The total number of pixel features, It can represent input features The spatial location of the corresponding feature map in a two-dimensional plane. First input set. The pixel features It is a 1×C dimensional vector. The value is an integer. In one example, a convolutional neural network may consist of a convolutional layer, a batch normalization layer, a ReLU activation layer, another convolutional layer, another convolutional layer, a batch normalization layer, and a ReLU activation layer in sequence.

[0070] A visual encoder is a deep learning model that uses learnable codewords as visual centers. In one example, a visual center codebook can include K visual centers (codewords or keys) and a set of smoothing factors. Each visual center is a 1×C dimensional codeword vector, connected to the first input set. Each pixel feature has the same dimension. First input set Each pixel feature is matched with K codeword vectors, and a smoothing factor can be used to adjust the smoothness of codeword matching. The visual encoder can learn a visual central codebook and apply it to the first input set. Encode the codeword and output the value corresponding to each codeword vector, which is the first positional feature. K is a positive integer.

[0071] For the first input set The pixel features and the Each codeword vector Based on the first input set The pixel features and the Each codeword vector The difference , or in combination with the first Each codeword vector Smoothing factor The difference after that can be generated by the neural network layer. Learnable weights . No. Learnable weights It can be represented as:

[0072] (4);

[0073] in, This represents the L2 parameter operation, also known as the squared penalty. It is a natural constant. Indicates the first One smoothing factor; Indicates the first Each codeword vector.

[0074] The first Learnable weights Multiply by the difference The first input set can be obtained. The pixel features Compared to the first The position of each codeword (visual center). Then, the entire first input set... Compared to the first The position of the first code character (visual center), i.e., the first... The first location feature corresponding to each visual center It can be represented as:

[0075] (5);

[0076] The first position features corresponding to multiple visual centers are fused to obtain the second position features. Second positional features It can be represented as:

[0077] (6);

[0078] in, and They are all 1×C dimensional vectors. This indicates a batch normalization (BN) layer containing the ReLU activation function.

[0079] Using a fully connected layer to incorporate second location features Mapped to C×1×1 dimension to be used as an influence factor. The first input set... With second position features Perform channel-level multiplication to obtain local region features. Local regional features It can be represented as:

[0080] (7);

[0081] in, Indicates a fully connected layer; This represents the Sigmoid activation function; This indicates channel-level multiplication.

[0082] The first input set and local regional features Add them together to obtain the first output feature. First output feature It can be represented as:

[0083] (8);

[0084] in, This indicates channel-level addition.

[0085] As shown in Figure 4, batch normalization layers, ReLU activation layers, and fully connected layers can be used for post-processing operations on the visual encoder. The first output features... The image is reconstructed to obtain a first reconstructed image, which can represent the first output feature. It mainly represents the feature information of the visual center.

[0086] According to embodiments of this application, a visual encoder can use a neural network to learn the center vectors (codeword vectors) of K visual centers end-to-end, and perform distance-weighted aggregation of the pixel features of the input features with the center vectors to obtain channel attention with global contextual semantics.

[0087] According to an embodiment of this application, the input features are grouped along the channel dimension to obtain a second input set; the second input set is input into a depth convolution unit to perform independent convolution operations on the features of each channel of the second input set, thereby capturing edge information and obtaining a convolved second input set; the convolved second input set is input into a channel multilayer perceptron unit to fuse the features of each channel of the convolved second input set to obtain a second output feature.

[0088] As shown in Figure 4, a multilayer perceptron may include convolutional layers, depth convolutional units, and channel multilayer perceptron units.

[0089] Input features can be processed along the channel dimension (channel C dimension) using convolutional layers. Grouping can be done, for example, by uniformly dividing along channel C. Group, to obtain the second input set : Second input set The Feature map It is p×H×W dimension, where, . It is an integer.

[0090] In one example, as shown in Figure 4, a deep convolutional unit may include a group normalization layer, a deep convolutional layer, a channel scaling layer, and a path dropout layer. The group normalization layer and the deep convolutional layer are used to process the second input set. The data is processed, including channel scaling and path discarding operations, and then compared with the second input set. Perform residual connections to obtain the second input set after convolution. The second input set after convolution. It can be represented as:

[0091] (9);

[0092] in, Indicates group normalization; This represents a depthwise convolution with a 1×1 kernel; Indicates channel scaling; This indicates that the path was discarded.

[0093] In one example, as shown in Figure 4, a channel multilayer perceptron unit may include a group normalization layer, a channel multilayer perceptron layer, a channel scaling layer, and a path dropping layer. The group normalization layer and the channel multilayer perceptron layer are used to process the convolutional second input set. The data is processed, including channel scaling and path discarding, and then convolved with the second input set. Perform residual connections to obtain the second output feature. Second output feature It can be represented as:

[0094] (10);

[0095] in, This refers to a multilayer sensor for the channel.

[0096] The second output feature The image is reconstructed to obtain a second reconstructed image, which can represent the second output feature. It mainly represents the characteristic information of the boundary.

[0097] According to embodiments of this application, input features are grouped along the channel dimension to obtain a second input set, which can improve feature representation while reducing computational cost. Processing the second input set using deep convolutional units and channel-based multilayer perceptron units can effectively reduce computational complexity and efficiently decode visual features. Employing channel scaling and path dropping operations can improve model generalization and robustness.

[0098] In the augmentation network, according to the embodiments of this application, each color channel of the augmentation-specific feature is inverted; illumination weights are applied to each inverted color channel and the layers are concatenated to obtain an attention weight map; texture features of the input image are extracted based on the local binary pattern features of the input image; and the attention weight map and texture features are concatenated to obtain the augmented output features.

[0099] Grayscale can reflect the lighting distribution of the entire image, but the degree to which lighting affects different color channels of the enhancement feature (image) is different. For example, the red color channel of the enhancement feature is more significantly affected by lighting, resulting in stronger contrast between light and dark; while the blue color channel of the enhancement feature is less affected, resulting in smaller differences between light and dark.

[0100] By performing a color inversion operation on each color channel of the enhanced features, bright areas are darkened and dark areas are brightened. This allows for more precise adjustment of the illumination-related features of each color channel, achieving better processing results. The inversion operation maps the intensity of each pixel in a single color channel to its complementary value in its corresponding color gamut, making bright areas darker and dark areas brighter. Different color channels of the enhanced features (image) are highly sensitive to changes in illumination intensity. After illumination constraints, the features of each color channel no longer fluctuate drastically with changes in illumination conditions, ensuring the accuracy of subsequent tasks such as classification, detection, and matching. For example, in face recognition, after applying illumination constraints to each of the three primary color channels of a face image, the extracted features remain consistent regardless of whether the face is in strong light, weak light, or sidelighting, preventing misjudgments due to different illumination levels.

[0101] The illumination weight of each color channel can be expressed as:

[0102] (11);

[0103] in, This indicates the location of the enhancement-specific feature on a single color channel. Illumination weights of pixels at that location. The position of the enhanced special feature after the inversion operation is indicated as The intensity of the pixel at that location; and These are the maximum and minimum values ​​of the pixel intensity after the inversion operation for enhancing the special features; and These are the maximum and minimum values ​​of the pixel intensity in the current color channel after the enhancement-specific feature has been inverted.

[0104] Illumination weights are applied to each of the inverted color channels to constrain lighting, and the resulting stitched images yield an attention weight map. .

[0105] (12);

[0106] in, This indicates that the three single color channels will be merged; This indicates that the enhanced special feature, after being inverted, is located in the red color channel at position 1. Illumination weights of pixels at that location; This indicates that the enhanced special feature, after being inverted, is located in the green color channel at the following position: Illumination weights of pixels at that location; This indicates that the enhanced special feature, after being inverted, is located in the blue color channel at the following position: The illumination weight of the pixel at that location. This represents the illumination weight of a pixel after the enhancement-specific feature has been inverted, resulting in the merging of the three single-color channels.

[0107] Based on the local binary pattern features of the input image, texture features are extracted. The input image is divided into multiple 16×16 small regions. For a pixel within each small region, the gray values ​​of its eight neighboring pixels are compared with the center pixel. A neighboring pixel's gray value is marked as 1 if it is greater than the center pixel's gray value, and marked as 0 if it is less than or equal to the center pixel's gray value. This generates an 8-bit binary number as the value of the center pixel for each small region. The frequency of each pixel's value within each small region is calculated, and the histogram is normalized. The histograms of each small region are concatenated to form a feature vector, which represents the texture features (Illumination-Invariant Local, ILL) of the entire input image.

[0108] The attention weight map and texture features are concatenated to obtain the enhanced output features.

[0109] According to embodiments of this application, by utilizing illumination constraints of color channels, the overall brightness of the image can be improved while preserving the details of the input image, thus preventing negative impacts on salient object detection tasks in low-light images due to detail loss and image blurring. Based on the local binary pattern of the input image, texture features are extracted from the input image, maintaining the stability of the texture features and improving the accuracy of edge detection in salient object detection tasks in low-light images.

[0110] As shown in Figure 3, the salient object detection network can include G dynamic multi-scale convolutions (DMC).

[0111] According to embodiments of this application, for the first dynamic multi-scale convolution, salient object detection-specific features are applied. Input the first dynamic multi-scale convolution to obtain the first rotation feature. For the i-th dynamic multi-scale convolution, the (i-1)-th rotation feature and the enhanced output feature are input into the i-th dynamic multi-scale convolution to obtain the i-th rotation feature. The i-th rotation feature is an integer 1 < i ≤ G, where the G-th rotation feature is the output of the salient object detection network.

[0112] According to an embodiment of this application, for the i-th dynamic multi-scale convolution, the (i-1)-th rotated feature... The input features of the i-th dynamic multi-scale convolution are concatenated with the enhanced output features to obtain the input features of the i-th dynamic multi-scale convolution. The (i-1)th rotational feature The image size for enhancing output features is consistent with the input features. The input features for the i-th dynamic multi-scale convolution... The image size is consistent with the input features.

[0113] Figure 5 shows a schematic diagram of dynamic multi-scale convolution according to an embodiment of this application.

[0114] As shown in Figure 5, in one example, dynamic multi-scale convolution can include dynamic blocks, multi-weight rotation layers, rotation weight layers, and convolutional layers.

[0115] Input features from the i-th dynamic multi-scale convolution Extracting multiple angular features and the weights of each of the multiple angular features .

[0116] (13);

[0117] in, This represents a dynamic block, where the convolutional kernel weights can be reparameterized based on the predicted angular features. Dynamic blocks can be used to extract orientation features and predict multiple angular features. and their respective weights In one example, as shown in Figure 5, a dynamic block may include a depthwise convolutional layer, a layer normalization layer, a ReLU activation layer, an adaptive average pooling layer, and a linear layer.

[0118] Based on the angle feature closest to the coordinates of each pixel among multiple angle features, an affine transformation is performed on the coordinates of each pixel to obtain the transformed coordinates of each pixel.

[0119] (14);

[0120] in, This represents the inverse matrix operation. Indicates the C angular features that are related to the first. coordinates of pixels The closest angular features Indicates by the first The angular features around the nth pixel affect the nth pixel. coordinates of pixels Perform an affine transformation to obtain the first... The transformed coordinates of each pixel .

[0121] Based on the transformed coordinates of each pixel, bilinear interpolation is performed on the convolution kernel weights of each pixel to obtain the interpolated convolution kernel weights of each pixel.

[0122] (15);

[0123] in, This represents bilinear interpolation. For the ... Convolution kernel weights of 1 pixel By performing bilinear interpolation, we can obtain the first... Interpolated convolution kernel weights for each pixel . No. The coordinates of the nth pixel change after the affine transformation. coordinates of pixels and the The transformed coordinates of each pixel There is a certain distance between them, the size of the feature map increases, and the weights of the convolution kernel change.

[0124] The orientation-aware features are obtained by weighting the interpolated convolution kernel weights of each pixel and the weights of the angle features closest to the coordinates of each pixel among multiple angle features.

[0125] Combine orientation-aware features with the input features of the i-th dynamic multi-scale convolution. Multiply to obtain the i-th rotation feature. .

[0126] (16);

[0127] Indicates the first The rotation weight of the nth pixel, The rotation weights of each pixel are determined by Corresponding weights and the Interpolated convolution kernel weights for each pixel Multiply them to get the result.

[0128] In one example, as shown in Figure 5, a multi-weighted rotation layer can be used based on the angular features of each pixel. and the weight of each pixel A bilinear interpolation method is used to perform multi-weight rotation operations on all pixels, obtaining the interpolated convolutional kernel weights for each pixel. The rotation weight layer can be used to sum the rotation weights of each pixel to generate orientation-aware features. These orientation-aware features are then combined with the input features of the i-th dynamic multi-scale convolution. Multiply to obtain the i-th rotation feature. Convolutional layers can be used to process the i-th rotated feature. Perform a convolution operation to adjust the i-th rotated feature. The scale information is used for subsequent operations.

[0129] In one example, the salient object detection network may include three dynamic multi-scale convolutions with input channel sizes of 320, 128, and 64, respectively.

[0130] According to an embodiment of this application, the (i-1)th rotation feature By concatenating the enhanced output features, negative transfer effects caused by detail loss and image blurring can be prevented in salient object detection tasks. In salient object detection networks, angular features are captured from salient object detection-specific features, and the convolutional kernel weights are dynamically reparameterized based on the angular features and weights to filter redundant features.

[0131] Figure 6 shows a schematic diagram of the detection results of a salient target according to an embodiment of this application.

[0132] As shown in Figure 6, the first column contains five low-light images, the second column contains the ground truth images of the five low-light images, the third column contains the detection results of salient targets obtained by processing the five low-light images using the ADMNet (Attention-guided Densely Multi-scale Network) image processing method, the fourth column contains the detection results of salient targets obtained by processing the five low-light images using the MINet (Multi-scale Interactive Network) image processing method, the fifth column contains the detection results of salient targets obtained by processing the five low-light images using the SI-SOD (Size-Invariant Salient Object Detection) image processing method, the sixth column contains the detection results of salient targets obtained by processing the five low-light images using the UGRAN (Uncertainty-Guided Refinement Attention Network) image processing method, and the seventh column contains the detection results of salient targets obtained by processing the five low-light images using the image processing method of the embodiment of this application. The image processing method of this application provides more accurate detection results for salient targets.

[0133] Figure 7 shows a schematic diagram of an image processing apparatus according to an embodiment of this application.

[0134] As shown in Figure 7, the image processing device 700 includes a preprocessing module 710, a decoupling module 720, an enhancement module 730, and a salient target detection module 740.

[0135] The preprocessing module 710 is used to preprocess the input image to obtain input features.

[0136] The decoupling module 720 is used to input the input features into the visual encoder and multilayer perceptron in the visual center decoupler to obtain enhancement-specific features and salient target detection-specific features. The visual encoder is used to aggregate local region features based on the input features to obtain enhancement-specific features, and the multilayer perceptron is used to capture edge information based on the input features to obtain salient target detection-specific features.

[0137] The enhancement module 730 is used to input enhancement-specific features into the enhancement network to obtain enhanced output features. The enhancement network uses the illumination weights of different color channels and the local binary pattern features of the input image as illumination constraints to enhance the enhancement-specific features, thereby obtaining enhanced output features.

[0138] The salient target detection module 740 is used to input salient target detection-specific features and enhanced output features into the salient target detection network to detect salient targets.

[0139] According to an embodiment of this application, the decoupling module 720 includes a visual encoder submodule, a multilayer sensor submodule, a stitching submodule, a first decoupling submodule, and a second decoupling submodule.

[0140] The visual encoder submodule is used to input the input features into the visual encoder to obtain the first output features.

[0141] The multilayer perceptron submodule is used to input the input features into the multilayer perceptron to obtain the second output features.

[0142] The splicing submodule is used to splice the first output feature and the second output feature to obtain the third output feature.

[0143] The first decoupling submodule is used to decouple the third output feature using the first convolutional network to obtain enhanced special features.

[0144] The second decoupling submodule is used to decouple the third output feature using the second convolutional network to obtain salient target detection-specific features.

[0145] According to embodiments of this application, the visual encoder submodule includes a mapping unit, a codebook encoding unit, a fusion unit, a multiplication unit, and an addition unit.

[0146] The mapping unit is used to map the input features pixel by pixel into multiple pixel features to obtain the first input set.

[0147] The codebook encoding unit is used to encode the first input set based on the visual center codebook to obtain the first position features corresponding to each of the multiple visual centers in the visual center codebook.

[0148] The fusion unit is used to fuse the first position features corresponding to multiple visual centers to obtain the second position features.

[0149] The multiplication unit is used to perform channel-level multiplication of the first input set and the second location feature to obtain local region features.

[0150] The addition unit is used to add the first input set and the local region features at the channel level to obtain the first output feature.

[0151] According to an embodiment of this application, the multilayer sensor submodule includes a grouping unit, a first convolutional unit, and a second convolutional unit.

[0152] The grouping unit is used to group the input features along the channel dimension to obtain the second input set.

[0153] The first convolutional unit is used to input the second input set into the depthwise convolutional unit to perform independent convolution operations on the features of each channel of the second input set, so as to obtain the convolved second input set.

[0154] The second convolutional unit is used to input the convolved second input set into the channel multilayer perceptron unit to fuse the features of each channel of the convolved second input set to obtain the second output feature.

[0155] According to an embodiment of this application, the enhancement module 730 includes a color inversion submodule, an illumination constraint submodule, a texture feature extraction submodule, and a feature splicing submodule.

[0156] The Color Inversion submodule is used to invert each color channel of the enhancement-specific feature.

[0157] The illumination constraint submodule is used to apply illumination weights to each inverted color channel and stitch them together to obtain an attention weight map.

[0158] The texture feature extraction submodule is used to extract texture features from the input image based on the local binary pattern features of the input image.

[0159] The feature concatenation submodule is used to concatenate the attention weight map and texture features to obtain enhanced output features.

[0160] According to embodiments of this application, the salient object detection network includes G dynamic multi-scale convolutions. The salient object detection module 740 includes a first rotation feature acquisition submodule and a second rotation feature acquisition submodule.

[0161] The first rotation feature acquisition submodule is used to input the salient object detection-specific features into the first dynamic multi-scale convolution to obtain the first rotation feature.

[0162] The second rotation feature acquisition submodule is used to input the (i-1)th rotation feature and the enhanced output feature into the i-th dynamic multi-scale convolution to obtain the i-th rotation feature.

[0163] An integer 1 < i ≤ G, where the Gth rotation feature is the output of the salient object detection network.

[0164] According to an embodiment of this application, the second rotation feature obtaining submodule includes a splicing unit, an extraction unit, a transformation unit, an interpolation unit, a weighted sum unit, and an obtaining unit.

[0165] The concatenation unit is used to concatenate the (i-1)th rotated feature and the enhanced output feature to obtain the input feature of the i-th dynamic multi-scale convolution.

[0166] The extraction unit is used to extract multiple angular features and their respective weights from the input features of the i-th dynamic multi-scale convolution.

[0167] The transformation unit is used to perform an affine transformation on the coordinates of each pixel based on the angular feature closest to the coordinates of each pixel among multiple angular features, so as to obtain the transformed coordinates of each pixel.

[0168] The interpolation unit is used to perform bilinear interpolation on the convolution kernel weights of each pixel based on the transformed coordinates of each pixel, so as to obtain the interpolated convolution kernel weights of each pixel.

[0169] The weighted sum unit is used to perform a weighted sum operation on the interpolated convolution kernel weights of each pixel and the weights of the angle features closest to the coordinates of each pixel among multiple angle features, to obtain the orientation-aware features.

[0170] The obtained unit is used to multiply the orientation-aware feature and the input feature of the i-th dynamic multi-scale convolution to obtain the i-th rotation feature.

[0171] According to an embodiment of this application, the preprocessing module 710 includes a first obtaining submodule and a second obtaining submodule.

[0172] The first submodule is used to input the input image into the backbone network and obtain the first and second features of the input image.

[0173] The second submodule is used to input the first and second features into the third convolutional network for feature fusion to obtain the input features.

[0174] According to embodiments of this application, any multiple modules among the preprocessing module 710, decoupling module 720, enhancement module 730, and salient target detection module 740 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the preprocessing module 710, decoupling module 720, enhancement module 730, and salient target detection module 740 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the preprocessing module 710, decoupling module 720, enhancement module 730, and salient target detection module 740 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0175] Figure 8 shows a block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of this application.

[0176] As shown in FIG8, an electronic device 800 according to an embodiment of the present application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0177] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0178] According to embodiments of this application, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0179] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0180] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0181] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the image processing method provided in the embodiments of this application.

[0182] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0183] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0184] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0185] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0186] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0187] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0188] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. An image processing method, characterized in that, The image processing method includes: preprocessing the input image to obtain input features; inputting the input features into a visual encoder and a multilayer perceptron in a visual center decoupler to obtain enhancement-specific features and salient object detection-specific features; the visual encoder is used to aggregate local region features based on the input features to obtain the enhancement-specific features, and the multilayer perceptron is used to capture edge information based on the input features to obtain the salient object detection-specific features; wherein, inputting the input features into the visual encoder and the multilayer perceptron in the visual center decoupler to obtain the enhancement-specific features and salient object detection-specific features includes: inputting the input features into the visual encoder to obtain a first output feature; inputting the input features into the multilayer perceptron to obtain a second output feature; and inputting the... The first output feature and the second output feature are concatenated to obtain the third output feature; the third output feature is decoupled using a first convolutional network to obtain the enhancement-specific feature; and the third output feature is decoupled using a second convolutional network to obtain the salient target detection-specific feature; the enhancement-specific feature is input into an enhancement network to obtain the enhancement output feature; wherein, the enhancement network is used to enhance the enhancement-specific feature by using the illumination weights of different color channels and the local binary pattern features of the input image as illumination constraints to obtain the enhancement output feature; wherein, the illumination weight is a coefficient that quantifies and controls the contribution of color channels to the overall illumination of the image; and the salient target detection-specific feature and the enhancement output feature are input into a salient target detection network to detect salient targets.

2. The image processing method according to claim 1, characterized in that, The step of inputting the input features into the visual encoder to obtain the first output feature includes: mapping the input features pixel by pixel into multiple pixel features to obtain a first input set; encoding the first input set based on a visual center codebook to obtain first position features corresponding to each of the multiple visual centers in the visual center codebook; fusing the first position features corresponding to each of the multiple visual centers to obtain a second position feature; performing channel-level multiplication of the first input set and the second position feature to obtain the local region feature; and performing channel-level addition of the first input set and the local region feature to obtain the first output feature.

3. The image processing method according to claim 1, characterized in that, The step of inputting the input features into the multilayer perceptron to obtain the second output feature includes: grouping the input features along the channel dimension to obtain a second input set; inputting the second input set into a depthwise convolutional unit to perform independent convolution operations on the features of each channel of the second input set to obtain a convolutional second input set; and inputting the convolutional second input set into a channel multilayer perceptron unit to fuse the features of each channel of the convolutional second input set to obtain the second output feature.

4. The image processing method according to claim 1, characterized in that, The step of inputting the enhancement-specific features into the enhancement network to obtain the enhancement output features includes: inverting each color channel of the enhancement-specific features; applying illumination weights to each inverted color channel and concatenating them to obtain an attention weight map; extracting the texture features of the input image based on the local binary pattern features of the input image; and concatenating the attention weight map and the texture features to obtain the enhancement output features.

5. The image processing method according to claim 1, characterized in that, The salient object detection network includes G dynamic multi-scale convolutions. The step of inputting the salient object detection-specific feature and the enhanced output feature into the salient object detection network to detect salient objects includes: for the first dynamic multi-scale convolution, inputting the salient object detection-specific feature into the first dynamic multi-scale convolution to obtain a first rotation feature; for the i-th dynamic multi-scale convolution, inputting the (i-1)-th rotation feature and the enhanced output feature into the i-th dynamic multi-scale convolution to obtain the i-th rotation feature; where 1 < i ≤ G, and the G-th rotation feature is the output of the salient object detection network.

6. The image processing method according to claim 5, characterized in that, The step of inputting the (i-1)th rotation feature and the enhanced output feature into the i-th dynamic multi-scale convolution to obtain the i-th rotation feature includes: concatenating the (i-1)th rotation feature and the enhanced output feature to obtain the input feature of the i-th dynamic multi-scale convolution; extracting multiple angle features and their respective weights from the input feature of the i-th dynamic multi-scale convolution; performing an affine transformation on the coordinates of each pixel based on the angle feature closest to the coordinates of each pixel among the multiple angle features to obtain the transformed coordinates of each pixel; performing bilinear interpolation on the convolution kernel weights of each pixel based on the transformed coordinates of each pixel to obtain the interpolated convolution kernel weights of each pixel; performing a weighted sum operation on the interpolated convolution kernel weights of each pixel and the weight of the angle feature closest to the coordinates of each pixel among the multiple angle features to obtain the orientation-aware feature; and multiplying the orientation-aware feature with the input feature of the i-th dynamic multi-scale convolution to obtain the i-th rotation feature.

7. The image processing method according to claim 1, characterized in that, The preprocessing of the input image to obtain input features includes: inputting the input image into a backbone network to obtain a first feature and a second feature of the input image; and inputting the first feature and the second feature into a third convolutional network for feature fusion to obtain the input features.

8. An image processing apparatus, characterized in that, The image processing device includes: a preprocessing module for preprocessing the input image to obtain input features; and a decoupling module for inputting the input features into a visual encoder and a multilayer perceptron in a visual center decoupler to obtain enhancement-specific features and salient target detection-specific features. The visual encoder is used to aggregate local region features based on the input features to obtain the enhancement-specific features, and the multilayer perceptron is used to capture edge information based on the input features to obtain the salient target detection-specific features. The decoupling module includes: a visual encoder submodule for inputting the input features into a visual encoder to obtain a first output feature; a multilayer perceptron submodule for inputting the input features into a multilayer perceptron to obtain a second output feature; and a stitching submodule for stitching the first and second output features together to obtain a third output feature. The system comprises: a first decoupling submodule, used to decouple the third output feature using a first convolutional network to obtain an enhanced specific feature; a second decoupling submodule, used to decouple the third output feature using a second convolutional network to obtain a salient target detection specific feature; an enhancement module, used to input the enhanced specific feature into an enhancement network to obtain an enhanced output feature; wherein the enhancement network uses the illumination weights of different color channels and the local binary pattern features of the input image as illumination constraints to enhance the enhanced specific feature to obtain the enhanced output feature; wherein the illumination weight is a coefficient that quantifies and controls the contribution of color channels to the overall illumination of the image; and a salient target detection module, used to input the salient target detection specific feature and the enhanced output feature into a salient target detection network to detect salient targets.

9. An electronic device, comprising: One or more processors; A memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the image processing method according to any one of claims 1 to 7.

Images