Salient target detection and depth prediction method and device and electronic equipment

Abstract

The invention provides a saliency target detection and depth prediction method and device and electronic equipment, and relates to the technical field of image processing, and the method comprises the steps: obtaining a target underwater image; inputting the target underwater image into the image detection prediction model to obtain a target salient image and a target depth image corresponding to the target underwater image; the image detection prediction model is used for aligning the features of the target underwater image with coordinates in a preset coordinate graph, and decoding an alignment feature graph after coordinate alignment based on a neighborhood integration strategy to obtain a target salient image and a target depth image; the alignment feature graph and the preset coordinate graph have the same spatial resolution. According to the method, high-resolution representation can be learned from low-resolution features, the saliency and the depth value of the corresponding position can be predicted, a high-quality prediction result can be generated under any specified resolution due to the fact that coordinates are continuous, and the edge detail recovery capacity and the overall prediction accuracy are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a salient target detection and depth prediction method, apparatus and electronic device. Background Technology

[0002] Underwater salient object detection (USOD) and underwater monocular depth estimation (UMDE) are two fundamental tasks in vision-based marine applications such as marine ecological monitoring, marine biometrics, unmanned underwater vehicle navigation, and underwater robot perception. USOD automatically identifies the most visually salient and semantically important regions from complex underwater images. UMDE uses a single RGB (Red, Green, Blue) image to predict the distance (i.e., depth information) from each pixel in the scene to the camera, providing essential 3D information for various vision-based underwater tasks.

[0003] Existing research on salient target detection and depth estimation methods mainly focuses on non-degraded natural images from land. These methods typically assume that the input image has good lighting conditions, rich texture information, and minimal imaging degradation. However, underwater environments present more severe challenges compared to natural scenes. Limited by the absorption and scattering effects of water, underwater images commonly suffer from color shifts, low contrast, blurring, non-uniform lighting, and occlusion. These factors significantly reduce image quality and increase the complexity of salient target detection and depth estimation tasks. Simultaneously, low-resolution images, due to their limited resolution, often suffer from missing details, blurred edges, reduced target scale, and insufficient feature representation, severely impacting the accuracy and reliability of salient target detection and monocular depth estimation. Most existing salient target detection and depth estimation methods are designed based on high-resolution images and are not specifically optimized for low-resolution images. This leads to inaccurate target localization, blurred boundaries, and large depth estimation errors in predictions on low-resolution underwater images, making them unsuitable for practical applications. Summary of the Invention

[0004] This invention provides a salient target detection and depth prediction method, apparatus, and electronic device to address the shortcomings of existing technologies in predicting low-resolution underwater images, such as inaccurate target localization, blurred boundaries, and large depth estimation errors.

[0005] This invention provides a method for salient target detection and depth prediction, comprising the following steps.

[0006] Acquire underwater images of the target.

[0007] The underwater image of the target is input into the image detection and prediction model to obtain the target salient image and target depth image corresponding to the underwater image of the target.

[0008] The image detection and prediction model is used to align the features of the underwater image of the target with the coordinates in the preset coordinate map, and decode the aligned feature map based on the neighborhood integration strategy to obtain the salient image of the target and the depth image of the target; the aligned feature map has the same spatial resolution as the preset coordinate map.

[0009] According to the salient target detection and depth prediction method provided by the present invention, the image detection prediction model includes a multi-scale feature extraction unit, a feature compression and optimization unit, a feature coordinate alignment unit, and a feature decoding unit; The step of inputting the underwater image of the target into the image detection and prediction model to obtain the target salient image and target depth image corresponding to the underwater image of the target includes: The underwater image of the target is input into the multi-scale feature extraction unit to obtain at least two first feature maps corresponding to each scale. The first feature maps corresponding to each of the at least two scales are input into the feature compression and optimization unit to obtain the second feature maps corresponding to each of the first feature maps. Input all the second feature maps into the feature coordinate alignment unit to obtain the aligned feature map and coordinate offset; The alignment feature map and the coordinate offset are input into the feature decoding unit to obtain the target salient image and the target depth image corresponding to the target underwater image.

[0010] According to the salient target detection and depth prediction method provided by the present invention, the step of inputting all second feature maps into the feature coordinate alignment unit to obtain aligned feature maps and coordinate offsets includes: Based on all the second feature maps, a concatenated feature map is obtained; The latent code in the spliced ​​feature map is determined to be closest to each query coordinate in the preset coordinate map. Based on the coordinates of each latent code and the corresponding query coordinates, the coordinate offset of each latent code is determined so as to align the alignment feature map with the coordinates of the preset coordinate map. The alignment feature map is determined based on all potential encodings.

[0011] According to the salient target detection and depth prediction method provided by the present invention, the step of inputting the aligned feature map and the coordinate offset into the feature decoding unit to obtain the salient image of the target and the depth image of the target corresponding to the underwater image of the target includes: Determine the normalized weights of the neighborhood latent codes corresponding to each query coordinate in the aligned feature map; Based on the alignment feature map, the coordinate offset, and each of the normalized weights, the target saliency value and target depth value corresponding to each query coordinate are obtained respectively. The target salient image is determined based on the target salient values ​​corresponding to all query coordinates; The target depth image is determined based on the target depth values ​​corresponding to all query coordinates.

[0012] According to the salient target detection and depth prediction method provided by the present invention, determining the normalized weights of the neighborhood latent codes corresponding to each query coordinate in the aligned feature map includes: Determine the area of ​​the diagonal rectangle corresponding to each of the neighborhood latent codes, and the total area of ​​the rectangle formed by all the neighborhood latent codes; the area of ​​the diagonal rectangle is the area of ​​the rectangle formed by the diagonal latent code on the diagonal direction of the neighborhood latent code and the query coordinate; Based on the area of ​​each diagonal rectangle and the area of ​​each total rectangle, the normalized weights corresponding to the potential codes of each neighborhood are determined.

[0013] According to the salient target detection and depth prediction method provided by the present invention, the step of inputting the first feature maps corresponding to the at least two scales into the feature compression and optimization unit to obtain the second feature maps corresponding to each of the first feature maps includes: Perform a convolution operation on each of the first feature maps to obtain a compressed feature map corresponding to each of the first feature maps; Based on the channel attention mechanism, the channel weights corresponding to each of the first feature maps are determined; Based on the compressed feature map and channel weights corresponding to each of the first feature maps, the second feature map corresponding to each of the first feature maps is determined.

[0014] According to the salient target detection and depth prediction method provided by the present invention, the step of obtaining a stitched feature map based on all second feature maps includes: Perform bilinear interpolation on all non-first second feature maps to obtain third feature maps corresponding to each non-first second feature map; each third feature map has the same resolution as the first second feature map; The first second feature map and all third feature maps are concatenated according to the channel dimension to obtain the concatenated feature map.

[0015] The present invention also provides a salient target detection and depth prediction device, comprising the following modules.

[0016] The acquisition module is used to acquire underwater images of the target.

[0017] The input module is used to input the underwater image of the target into the image detection and prediction model to obtain the target salient image and the target depth image corresponding to the underwater image of the target.

[0018] The image detection and prediction model is used to align the features of the underwater image of the target with the coordinates in the preset coordinate map, and decode the aligned feature map based on the neighborhood integration strategy to obtain the salient image of the target and the depth image of the target; the aligned feature map has the same spatial resolution as the preset coordinate map.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the salient target detection and depth prediction method as described above.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the salient target detection and depth prediction methods as described above.

[0021] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the salient target detection and depth prediction method as described above.

[0022] The present invention provides a salient target detection and depth prediction method, apparatus, and electronic device. By inputting an acquired underwater image of the target into an image detection and prediction model, the model extracts features from the underwater image and aligns these features with coordinates in a preset coordinate map. Based on a neighborhood ensemble strategy, the aligned feature map, with the same spatial resolution as the preset coordinate map, is decoded to obtain a salient target image and a depth image. In this embodiment, by introducing an implicit image representation mechanism, continuous image coordinates and surrounding local features are used as input. A high-resolution representation is learned from low-resolution features to predict the saliency and depth values ​​at corresponding locations, thus overcoming the limitations of traditional discrete pixel grids. Since the coordinates are continuous, this invention can generate high-quality prediction results at any specified resolution, significantly improving the ability to recover edge details and the overall prediction accuracy. Furthermore, by simultaneously predicting the salient target image and depth image through a unified network structure, the robustness and accuracy of the underwater image perception system are significantly improved by fully utilizing the complementarity between saliency and depth information. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the salient target detection and depth prediction method provided in this embodiment of the invention.

[0025] Figure 2 This is a schematic diagram of the structure of the image detection and prediction model provided in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the location of the neighborhood potential encoding of the query coordinates provided in the embodiments of the present invention.

[0027] Figure 4 This is a flowchart illustrating the training method of the image detection and prediction model provided in this embodiment of the invention.

[0028] Figure 5 This is a comparative schematic diagram of the high-resolution saliency prediction results provided in the embodiments of the present invention.

[0029] Figure 6 This is a comparative schematic diagram of the high-resolution depth map prediction results provided in the embodiments of the present invention.

[0030] Figure 7 This is a schematic diagram of the saliency map prediction results at arbitrary resolution provided in the embodiments of the present invention.

[0031] Figure 8 This is a schematic diagram of the arbitrary resolution depth map prediction results provided in the embodiments of the present invention.

[0032] Figure 9 This is a schematic diagram of the salient target detection and depth prediction device provided in an embodiment of the present invention.

[0033] Figure 10 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] To address the problems of inaccurate target localization, blurred boundaries, and large depth estimation errors in existing technologies for predicting low-resolution underwater images, this invention provides a salient target detection and depth prediction method. Figure 1 This is a flowchart illustrating the salient target detection and depth prediction method provided in this embodiment of the invention, as shown below. Figure 1 As shown, the method includes steps 110 and 120.

[0036] Step 110: Obtain underwater images of the target.

[0037] It should be noted that the underwater image of the target can be a low-resolution image acquired by an underwater vehicle, underwater robot, or other underwater camera equipment. For example, the resolution of the underwater image of the target can be 96×96. In resource-constrained application scenarios, this low-resolution image can reduce computational load and lower storage and transmission costs.

[0038] Step 120: Input the underwater image of the target into the image detection and prediction model to obtain the target salient image and target depth image corresponding to the underwater image of the target; The image detection and prediction model is used to align the features of the underwater image of the target with the coordinates in the preset coordinate map, and decode the aligned feature map based on the neighborhood integration strategy to obtain the salient image of the target and the depth image of the target; the aligned feature map has the same spatial resolution as the preset coordinate map.

[0039] Specifically, in existing technologies, underwater salient target detection and monocular depth estimation are typically treated as two independent tasks, lacking in-depth exploration and utilization of their potential correlation. In reality, salient target detection can provide prior information on foreground and background distribution for depth estimation, while depth information can help salient target detection more accurately understand the spatial structure of the scene and object boundaries. Therefore, ignoring the complementary relationship between the two may lead to insufficient feature utilization, limiting the model's perception performance and overall effectiveness in complex underwater environments. Therefore, in this embodiment of the invention, after acquiring the target underwater image, the target underwater image is input into a pre-trained image detection prediction model. This image detection prediction model does not directly predict on a discrete pixel grid, but instead transforms the decoding process of features in the target underwater image into a query process in a continuous coordinate space corresponding to a preset coordinate map. Specifically, the process is as follows: First, features are extracted from the low-resolution target underwater image. The extracted features are then aligned with the coordinates in a preset coordinate map with a preset spatial resolution, thus mapping the discrete features to a continuous coordinate space and obtaining an aligned feature map with the same spatial resolution as the preset coordinate map. This effectively upscales the extracted features to the spatial resolution corresponding to the preset coordinate map. Subsequently, a neighborhood ensemble strategy is employed to fuse information from multiple features surrounding the query coordinates, and the aligned feature map is decoded to obtain the target salient image and target depth image corresponding to the preset coordinate map. This ensures the continuity of edges and the accuracy of detailed features in the generated target salient and target depth images. The target salient image is used to characterize the most visually salient and semantically important region in the underwater image of the target, while the target depth image is used to characterize the distance from each pixel to the camera, providing necessary 3D information for various vision-based underwater tasks. The target salient and target depth images are complementary, mutually enhancing feature representation capabilities. For example, the target depth image provides additional geometric information for salient target detection corresponding to the target salient image, and provides foreground and background cues for monocular depth estimation corresponding to the target depth image.

[0040] The salient target detection and depth prediction method provided in this invention involves inputting an acquired underwater image of the target into an image detection and prediction model. The model extracts features from the underwater image and aligns these features with coordinates in a preset coordinate map. Based on a neighborhood ensemble strategy, the aligned feature map, with the same spatial resolution as the preset coordinate map, is decoded to obtain the target salient image and target depth image. In this invention, an implicit image representation mechanism is introduced. Using continuous image coordinates and surrounding local features as input, a high-resolution representation is learned from low-resolution features to predict the salientity and depth values ​​at corresponding locations. This overcomes the limitations of traditional discrete pixel grids. Because the coordinates are continuous, this invention can generate high-quality prediction results at any specified resolution, significantly improving the ability to recover edge details and the overall prediction accuracy. Furthermore, by using a unified network structure to simultaneously predict the target salient image and target depth image, the robustness and accuracy of the underwater image perception system are significantly improved by fully utilizing the complementarity between salientity and depth information.

[0041] In one embodiment, Figure 2 This is a schematic diagram of the structure of the image detection and prediction model provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the image detection prediction model includes a multi-scale feature extraction unit, a feature compression and optimization unit, a feature coordinate alignment unit, and a feature decoding unit.

[0042] The step of inputting the underwater image of the target into the image detection and prediction model to obtain the target salient image and target depth image corresponding to the underwater image of the target includes: The underwater image of the target is input into the multi-scale feature extraction unit to obtain at least two first feature maps corresponding to each scale. The first feature maps corresponding to each of the at least two scales are input into the feature compression and optimization unit to obtain the second feature maps corresponding to each of the first feature maps. Input all the second feature maps into the feature coordinate alignment unit to obtain the aligned feature map and coordinate offset; The alignment feature map and the coordinate offset are input into the feature decoding unit to obtain the target salient image and the target depth image corresponding to the target underwater image.

[0043] Specifically, the image detection and prediction model comprises four functionally decoupled and sequentially cascaded units: a multi-scale feature extraction unit, a feature compression and optimization unit, a feature coordinate alignment unit, and a feature decoding unit. After the target underwater image is input into the image detection and prediction model, the model decomposes the complex task of underwater salient target detection and monocular depth prediction into four independent and sequential steps: feature extraction, feature compression and optimization, alignment, and feature decoding. The specific steps include the following:

[0044] (1) The acquired underwater target image is input into a multi-scale feature extraction unit, which includes n encoders, where n is an integer greater than 1. Different encoders are used to extract the first feature maps of the underwater target image at different scales to capture the feature information of the underwater target image at different levels of abstraction. Taking n as 4 as an example, the extraction process of the first feature maps at 4 different scales can be represented as F={F1,F2,F3,F4}=E(x), where E represents the encoder; x represents the low-resolution underwater target image, F i Let i represent the i-th first feature map extracted by the i-th encoder, where 1 ≤ i ≤ n.

[0045] For example, such as Figure 2 As shown, n encoders are connected in series. That is, the first feature map at the i-th scale extracted by the i-th encoder is used as the input data of the (i+1)-th encoder. The (i+1)-th encoder is used to downsample the first feature map at the i-th scale to extract the first feature map at the (i+1)-th scale. The i-th scale is different from the (i+1)-th scale.

[0046] Furthermore, due to light scattering and absorption, underwater images of targets are often affected by quality degradation such as color shift, contrast distortion, and fogging effects. To address this degradation, the multi-scale feature extraction unit can employ a multi-scale visual Transformer as its backbone network. The multi-scale visual Transformer utilizes a multi-head attention mechanism, exhibiting stronger robustness and performance than convolutional neural networks (CNNs). This invention does not limit the type of backbone network.

[0047] (2) After obtaining the first feature maps at different scales, the multiple first feature maps are input into the feature compression and optimization unit. The multiple first feature maps are subjected to channel dimensionality reduction and feature enhancement based on channel attention mechanism in the feature compression and optimization unit to obtain the optimized second feature maps at different scales.

[0048] (3) After obtaining the optimized second feature maps at different scales, the multiple second feature maps are input into the feature coordinate alignment unit. The multiple second feature maps are fused in the feature coordinate alignment unit and aligned with a coordinate space of a preset spatial resolution to obtain the aligned feature map and coordinate offset.

[0049] (4) After obtaining the alignment feature map and coordinate offset, the feature decoding unit uses the alignment feature map and coordinate offset to calculate the target saliency value and depth value corresponding to each query coordinate through the neighborhood integration strategy, thereby obtaining the target saliency image and target depth image of the underwater image of the target.

[0050] In this embodiment, the phased processing flow ensures the orderliness and efficiency of salient object detection and monocular depth prediction. Each unit focuses on a specific subtask, and the overall image detection and prediction model has a clear structure that is easy to train and optimize.

[0051] In one embodiment, inputting the first feature maps corresponding to each of the at least two scales into the feature compression and optimization unit to obtain the second feature maps corresponding to each of the first feature maps includes: Perform a convolution operation on each of the first feature maps to obtain a compressed feature map corresponding to each of the first feature maps; Based on the channel attention mechanism, the channel weights corresponding to each of the first feature maps are determined; Based on the compressed feature map and channel weights corresponding to each of the first feature maps, the second feature map corresponding to each of the first feature maps is determined.

[0052] Specifically, since salient object detection and monocular depth prediction require both pixel-level and semantic-level information, in this embodiment of the invention, after obtaining the first feature maps at all scales, two parallel operations are performed on each first feature map. One operation involves convolving the first feature maps at each scale using 3×3 convolutional blocks to obtain compressed feature maps corresponding to each first feature map, thus projecting the number of channels of all first feature maps to a target dimension, for example, 32. The other operation involves inputting each first feature map into a channel attention subunit, and calculating the channel weights corresponding to each first feature map by analyzing the global information of each channel in each first feature map. Then, the product of each compressed feature map and its corresponding channel weight is calculated, and this product is the optimized second feature map. This second feature map is represented as: Z i =Conv(F i )×W(F i ), where Conv represents 3×3 convolution; W represents channel attention mechanism; Z i Let i represent the i-th second feature map.

[0053] In this embodiment of the invention, channel compression effectively reduces the number of model parameters and computational cost. Channel attention mechanism reduces model complexity and balances the influence between low-level and high-level features, significantly improving the signal-to-noise ratio and effectiveness of feature representation, and providing high-quality input data for subsequent feature fusion and alignment.

[0054] In one embodiment, the step of inputting all the second feature maps into the feature coordinate alignment unit to obtain the aligned feature map and coordinate offset includes: Based on all the second feature maps, a concatenated feature map is obtained; The latent code in the spliced ​​feature map is determined to be closest to each query coordinate in the preset coordinate map. Based on the coordinates of each latent code and the corresponding query coordinates, the coordinate offset of each latent code is determined so as to align the alignment feature map with the coordinates of the preset coordinate map. The alignment feature map is determined based on all potential encodings.

[0055] Specifically, in existing technologies, linear interpolation (e.g., bilinear interpolation and bicubic interpolation) combined with convolution operations are typically used to upsample features to a specified resolution. These methods tend to blur the fine contextual information of the features, thus reducing the performance of subsequent tasks. Furthermore, these methods cannot generate prediction results of arbitrary sizes, limiting the flexibility of the network in different application scenarios. Therefore, in this embodiment of the invention, after obtaining each second feature map, all second feature maps are fused to obtain a stitched feature map, and the stitched feature map is aligned with the coordinates in a preset coordinate map. This allows the salient object detection and monocular depth prediction processes to learn continuous feature representations for decoding at arbitrary spatial resolutions. Specifically, the following steps are included: (1) In order to learn continuous feature representations, a two-dimensional preset coordinate graph is first created according to the preset spatial resolution. Each image coordinate in the preset coordinate graph represents a query coordinate.

[0056] (2) Each feature vector in the stitched feature map is considered as a latent code uniformly distributed in two-dimensional space. For each query coordinate, the feature vector closest to the query coordinate is found in the stitched feature map, and this feature vector is the latent code closest to the query coordinate. At the same time, the coordinates of the latent code in the stitched feature map are calculated, and the difference between the query coordinate and the coordinates of the latent code is calculated. This difference is the coordinate offset, which records the precise sub-pixel position information of the latent code. After determining the latent codes corresponding to all query coordinates, all latent codes form an aligned feature map. This can be understood as filling each latent code into the corresponding position of a blank feature map with the same resolution as the preset coordinate map. The filled blank feature map is the aligned feature map, and the spatial resolution of the aligned feature map is the same as that of the preset coordinate map. The feature coordinate alignment can be represented as: {z', x q -x'}=g(x q ,Z), where g represents the feature coordinate alignment function; x q Z represents the query coordinates in the preset coordinate map; Z represents the aligned feature map; z' represents the latent code in the aligned feature map Z that is closest to the query coordinates; x' represents the coordinates corresponding to the latent code z'; x q -x' represents the coordinate offset.

[0057] Existing technologies, such as linear interpolation and convolution, expand feature maps by weighted averaging of the features themselves, which is essentially a way of smoothing and blurring information. In contrast, this invention does not alter the feature vectors themselves. Instead, it uses coordinate lookup and rearrangement to transfer the original, information-rich latent encoding to a network of arbitrary resolution. This enhances the expressive power of low-resolution feature maps to any target resolution in a way that preserves information, facilitating the generation of clear and detailed images. Simultaneously, the coordinate offset facilitates the perception of sub-pixel level deviations. This is an information-preserving upsampling mechanism that helps restore fine structures and sharp edges.

[0058] It should be noted that the preset spatial resolution can be set according to user needs. For example, the preset spatial resolution can be 96×96, 192×192, etc. The embodiments of the present invention do not limit this.

[0059] In one embodiment, obtaining the concatenated feature map based on all the second feature maps includes: Perform bilinear interpolation on all non-first second feature maps to obtain third feature maps corresponding to each non-first second feature map; each third feature map has the same resolution as the first second feature map; The first second feature map and all third feature maps are concatenated according to the channel dimension to obtain the concatenated feature map.

[0060] For example, taking four second feature maps as an example, since the scales of each second feature map are different, in this embodiment of the invention, the second feature map Z1 has the highest spatial resolution. Bilinear interpolation is performed on the second feature maps Z2, Z3, and Z4, which have lower spatial resolutions, to enlarge their spatial resolutions to the same level as the second feature map Z1, resulting in third feature maps Y2, Y3, and Y4. Then, the second feature maps Z1, Y2, Y3, and Y4 are stitched together along the channel dimension to obtain the stitched feature map Z.

[0061] In this embodiment, by upsampling the non-first second feature maps with lower spatial resolution, spatial resolution alignment of all second feature maps is achieved. This allows high-level semantic information (e.g., location and category information) and low-level detail information (e.g., precise edges and texture information) from different scales to be aggregated into a single feature representation. This information complementarity is crucial for the subsequent simultaneous completion of salient object detection and monocular depth prediction, ensuring that the final output accurately locates the target and finely characterizes boundary and depth information.

[0062] In one embodiment, inputting the aligned feature map and the coordinate offset into the feature decoding unit to obtain the target salient image and the target depth image corresponding to the target underwater image includes: Determine the normalized weights of the neighborhood latent codes corresponding to each query coordinate in the aligned feature map; Based on the alignment feature map, the coordinate offset, and each of the normalized weights, the target saliency value and target depth value corresponding to each query coordinate are obtained respectively. The target salient image is determined based on the target salient values ​​corresponding to all query coordinates; The target depth image is determined based on the target depth values ​​corresponding to all query coordinates.

[0063] Specifically, for each query coordinate, based on the relative position of the query coordinate with the four neighboring latent codes, the normalized weights corresponding to each neighboring latent code are calculated. The four neighboring latent codes are feature vectors located in the four neighboring positions (top left, top right, bottom left, and bottom right) of the query coordinate in the aligned feature map. The closer the query coordinate is to a neighboring latent code, the larger the normalized weight of that neighboring latent code. Next, this feature decoding unit includes a first decoder and a second decoder. The first decoder is used for salient object detection, and the second decoder is used for monocular depth prediction. The aligned feature map, coordinate offset, and the normalized weights corresponding to each query coordinate are input into the first decoder for salient object detection, and the output is the target saliency value corresponding to each query coordinate, fused according to the normalized weights. Simultaneously, the aligned feature map, coordinate offset, and the normalized weights corresponding to each query coordinate are input into the second decoder for monocular depth prediction, and the output is the target depth value corresponding to each query coordinate.

[0064] The saliency value or depth value of the target can be expressed as: Among them, M j Let M represent the j-th predicted value, where j = [1, 2]. When j = 1, M... j M represents the target saliency value output by the first decoder, when j=2. j This represents the target depth value output by the second decoder; k represents the location code of the neighborhood latent code, for example, k=00 indicates that the neighborhood latent code is located at the query coordinate x. q The top-left position, k=01 indicates that the neighborhood latent encoding is at the query coordinate x. q The upper right position, k=10 indicates that the neighborhood latent encoding is at the query coordinate x. q The lower left position, k=11 indicates that the neighborhood latent encoding is at the query coordinate x. q The lower right position; S k Let S represent the area of ​​the diagonal rectangle corresponding to the latent code in the neighborhood at position k, that is, the area of ​​the diagonal rectangle formed by the diagonal latent code in the diagonal direction of the latent code in the neighborhood at position k and the query coordinate; S represents the total area of ​​the rectangle formed by the four latent codes in the neighborhood, that is, ; Represents the latent encoding at position k The corresponding normalized weights; G j This represents the predicted sub-value output by the j-th decoder; This represents the potential encoding of the neighborhood at position k; Representing the latent encoding of the neighborhood The coordinates of the location; Representing the latent encoding of the neighborhood The corresponding coordinate offset; Representing the latent encoding of the neighborhood The independently predicted values ​​are summed by summing the predicted values ​​of the four neighborhood latent codes to obtain the final target predicted value (i.e., target salience value or target depth value) for the query coordinates.

[0065] Subsequently, the target saliency values ​​corresponding to all query coordinates constitute the final target saliency image, and the target depth values ​​corresponding to all query coordinates constitute the final target depth image.

[0066] It should be noted that both the first and second decoders are composed of four-layer Multilayer Perceptrons (MLPs), and the activation function can be the ReLU (Rectified Linear Unit) function. The hidden layer dimension of the MLP can be 128, and the output dimension is 1.

[0067] In this embodiment, by smoothly fusing information from multiple neighborhood feature vectors around each query coordinate, the continuity and smoothness of the output image between pixels are ensured, thereby generating a high-quality image with sharp and continuous edges.

[0068] In one embodiment, determining the normalized weights of the neighborhood latent encoding corresponding to each query coordinate in the aligned feature map includes: Determine the area of ​​the diagonal rectangle corresponding to each of the neighborhood latent codes, and the total area of ​​the rectangle formed by all the neighborhood latent codes; the area of ​​the diagonal rectangle is the area of ​​the rectangle formed by the diagonal latent code on the diagonal direction of the neighborhood latent code and the query coordinate; Based on the area of ​​each diagonal rectangle and the area of ​​each total rectangle, the normalized weights corresponding to the potential codes of each neighborhood are determined.

[0069] Specifically, the query coordinates are located within a rectangular unit consisting of four neighboring latent codes, with each neighboring latent code and the query coordinates forming a rectangle, and the four rectangles constituting a total rectangle. For each neighboring latent code, the diagonal latent code and the query coordinates form a diagonal rectangle. The ratio of the area of ​​the diagonal rectangle to the area of ​​the total rectangle is the normalized weight corresponding to that neighboring latent code; the larger the area of ​​the diagonal rectangle, the larger the normalized weight.

[0070] For example, Figure 3 This is a schematic diagram of the location of the neighborhood latent encoding of the query coordinates provided in an embodiment of the present invention, such as... Figure 3 As shown, the query coordinates are surrounded by four neighborhood latent codes, where the neighborhood latent code z' 00 Located at the top left of the query coordinates, the neighborhood latent encoding z' 01 Located at the top right of the query coordinates, the neighborhood latent encoding z' 10Located at the lower left of the query coordinates, the neighborhood latent encoding z' 11 It is located at the bottom right of the query coordinates. The neighborhood latent encoding z' is... 11 For the neighborhood latent encoding z' 00 The diagonal latent encoding, S 00 For the neighborhood latent encoding z' 11 The area of ​​the rectangle formed by the query coordinates and the neighborhood latent code z' 00 The area of ​​the diagonal rectangle. Similarly, the neighborhood latent encoding z' 10 For the neighborhood latent encoding z' 01 The diagonal latent encoding, S 01 For the neighborhood latent encoding z' 10 The area of ​​the rectangle formed by the query coordinates and the neighborhood latent code z' 01 The area of ​​the diagonal rectangle. Neighborhood latent encoding z' 01 For the neighborhood latent encoding z' 10 The diagonal latent encoding, S 10 For the neighborhood latent encoding z' 01 The area of ​​the rectangle formed by the query coordinates and the neighborhood latent code z' 10 The area of ​​the diagonal rectangle. Neighborhood latent encoding z' 00 For the neighborhood latent encoding z' 11 The diagonal latent encoding, S 11 For the neighborhood latent encoding z' 00 The area of ​​the rectangle formed by the query coordinates and the neighborhood latent code z' 11 The area of ​​the diagonal rectangle. The total area of ​​the rectangle S = S 00 +S 01 +S 10 +S 11 Therefore, the neighborhood latent encoding z' 00 The corresponding normalized weight is S 00 / S, neighborhood latent encoding z' 01 The corresponding normalized weight is S 01 / S, neighborhood latent encoding z' 10 The corresponding normalized weight is S 10 / S, neighborhood latent encoding z' 11 The corresponding normalized weight is S 11 Therefore, with the total rectangle area remaining constant, the closer the neighborhood latent encoding is to the query coordinates, the larger the area of ​​the diagonal rectangle of the neighborhood latent encoding, and the larger the corresponding normalization weight.

[0071] In this embodiment, the neighborhood integration strategy is integrated into the decoding process to avoid the problem of discontinuous edges in the prediction results.

[0072] It should be noted that, Figure 4This is a flowchart illustrating the training method of the image detection and prediction model provided in this embodiment of the invention, as shown below. Figure 4 As shown, the image detection prediction model in this embodiment of the invention is trained based on the following steps.

[0073] (1) Randomly initialize the network parameters of the initial image detection prediction model.

[0074] (2) Randomly select training samples from the training dataset and input them into the initial image detection prediction model.

[0075] (3) Use multi-scale visual Transformer for multi-scale feature extraction.

[0076] (4) Based on the channel attention mechanism, the extracted features at different scales are compressed and optimized.

[0077] (5) Use the hermit representation method to align the optimized multi-scale features with the coordinates.

[0078] (6) Two MLP networks are used to predict the saliency map and depth map respectively.

[0079] (7) Determine whether the initial image detection prediction model has converged. If it has not converged, update the network parameters of the initial image detection prediction model and jump to step (2) for the next round of iterative training; if it has converged, end the training and obtain the trained image detection prediction model.

[0080] It should be noted that this embodiment of the invention is trained and tested on the USOD10K dataset. The USOD10K dataset contains 10255 underwater RGB images, along with finely annotated saliency maps and depth maps. This USOD10K dataset is divided into 9229 training images and 1026 test images. During the training phase, the training images, saliency maps, and depth maps are uniformly adjusted to a resolution of 96×96. The Adam optimizer is used during training, with an initial learning rate of 0.0001, a batch size of 16, and a training duration of 200 epochs. The learning rate is reduced to 0.5 times the original value every 50 epochs. During the testing phase, low-resolution underwater images (96×96) are used as input to generate saliency maps and depth maps of arbitrary sizes. During training, this embodiment of the invention employs a supervised learning loss function to guide network training. Specifically, weighted IoU loss and weighted BCE loss are used to guide the training of saliency maps, while mean-square error (MSE) and structural similarity index (SSIM) loss are used to guide the training of depth maps.

[0081] It should be noted that the salient target detection and depth prediction method provided in this embodiment of the invention can more accurately predict the salient map and depth map of low-resolution underwater images compared with existing technologies. For example, Figure 5 This is a comparative schematic diagram of the high-resolution saliency prediction results provided in the embodiments of the present invention, such as... Figure 5 As shown, the saliency map predicted by the embodiments of the present invention has clearer edge contours and can more accurately locate key regions in the image. Its results are highly consistent with the ground truth (GT) saliency map and significantly outperform existing methods such as U2Net (a deep learning model for salient object detection) and ICON. For example, Figure 6 This is a comparative schematic diagram of the high-resolution depth map prediction results provided in the embodiments of the present invention, such as... Figure 6 As shown, in terms of depth map prediction, compared with existing methods DPT (Depth Pixel Transport) and NeWCRFs (Neural Window Fully-connected Conditional Random Fields), the high-resolution depth map generated by the embodiments of the present invention has significant advantages in maintaining detail level and structural integrity. The recovered scene structure is more realistic and coherent, and closer to the real depth map (i.e., Figure 6 The distribution characteristics of GT are shown in the figure. For example, Figure 7 This is a schematic diagram of the saliency map prediction results at arbitrary resolution provided in an embodiment of the present invention. Figure 8 This is a schematic diagram of the arbitrary resolution depth map prediction results provided in the embodiments of the present invention, such as... Figure 7 and Figure 8 As shown, this invention overcomes the limitations of existing methods on input image resolution, achieving joint prediction of saliency maps and depth maps for underwater images of arbitrary resolution. Experimental results demonstrate that this invention can output high-quality saliency maps and depth maps of arbitrary resolution even with low-resolution input images, fully showcasing the technical advantages of the method provided in terms of resolution adaptability and cross-scale representation capabilities. In summary, this invention significantly outperforms existing technologies in terms of accuracy, robustness, structural integrity, and resolution universality, possessing high practical application value and promising prospects for widespread adoption.

[0082] The salient target detection and depth prediction apparatus provided by the present invention will be described below. The salient target detection and depth prediction apparatus described below can be referred to in correspondence with the salient target detection and depth prediction method described above.

[0083] This invention also provides a salient target detection and depth prediction device. Figure 9This is a schematic diagram of the salient target detection and depth prediction device provided in an embodiment of the present invention, as shown below. Figure 9 As shown, the salient target detection and depth prediction device 900 includes an acquisition module 910 and an input module 920.

[0084] The acquisition module 910 is used to acquire underwater images of the target.

[0085] The input module 920 is used to input the target underwater image into the image detection and prediction model to obtain the target salient image and target depth image corresponding to the target underwater image.

[0086] The image detection and prediction model is used to align the features of the underwater image of the target with the coordinates in the preset coordinate map, and decode the aligned feature map based on the neighborhood integration strategy to obtain the salient image of the target and the depth image of the target; the aligned feature map has the same spatial resolution as the preset coordinate map.

[0087] The salient target detection and depth prediction device provided in this invention inputs the acquired underwater image of the target into an image detection and prediction model. The model extracts features from the underwater image and aligns these features with coordinates in a preset coordinate map. Based on a neighborhood ensemble strategy, it decodes the aligned feature map, which has the same spatial resolution as the preset coordinate map, to obtain the target salient image and the target depth image. In this invention, by introducing an implicit image representation mechanism, continuous image coordinates and surrounding local features are used as input. A high-resolution representation is learned from low-resolution features to predict the salientity and depth values ​​at corresponding locations, thus overcoming the limitations of traditional discrete pixel grids. Because the coordinates are continuous, this invention can generate high-quality prediction results at any specified resolution, significantly improving the ability to recover edge details and the overall prediction accuracy. Furthermore, by simultaneously predicting the target salient image and the target depth image through a unified network structure, the robustness and accuracy of the underwater image perception system are significantly improved by fully utilizing the complementarity between salientity and depth information.

[0088] Optionally, the image detection prediction model includes a multi-scale feature extraction unit, a feature compression and optimization unit, a feature coordinate alignment unit, and a feature decoding unit.

[0089] Optionally, the input module 920 is specifically used for: The underwater image of the target is input into the multi-scale feature extraction unit to obtain at least two first feature maps corresponding to each scale. The first feature maps corresponding to each of the at least two scales are input into the feature compression and optimization unit to obtain the second feature maps corresponding to each of the first feature maps. Input all the second feature maps into the feature coordinate alignment unit to obtain the aligned feature map and coordinate offset; The alignment feature map and the coordinate offset are input into the feature decoding unit to obtain the target salient image and the target depth image corresponding to the target underwater image.

[0090] Optionally, the input module 920 is specifically used for: Based on all the second feature maps, a concatenated feature map is obtained; The latent code in the spliced ​​feature map is determined to be closest to each query coordinate in the preset coordinate map. Based on the coordinates of each latent code and the corresponding query coordinates, the coordinate offset of each latent code is determined so as to align the alignment feature map with the coordinates of the preset coordinate map. The alignment feature map is determined based on all potential encodings.

[0091] Optionally, the input module 920 is specifically used for: Determine the normalized weights of the neighborhood latent codes corresponding to each query coordinate in the aligned feature map; Based on the alignment feature map, the coordinate offset, and each of the normalized weights, the target saliency value and target depth value corresponding to each query coordinate are obtained respectively. The target salient image is determined based on the target salient values ​​corresponding to all query coordinates; The target depth image is determined based on the target depth values ​​corresponding to all query coordinates.

[0092] Optionally, the input module 920 is specifically used for: Determine the area of ​​the diagonal rectangle corresponding to each of the neighborhood latent codes, and the total area of ​​the rectangle formed by all the neighborhood latent codes; the area of ​​the diagonal rectangle is the area of ​​the rectangle formed by the diagonal latent code on the diagonal direction of the neighborhood latent code and the query coordinate; Based on the area of ​​each diagonal rectangle and the area of ​​each total rectangle, the normalized weights corresponding to the potential codes of each neighborhood are determined.

[0093] Optionally, the input module 920 is specifically used for: Perform a convolution operation on each of the first feature maps to obtain a compressed feature map corresponding to each of the first feature maps; Based on the channel attention mechanism, the channel weights corresponding to each of the first feature maps are determined; Based on the compressed feature map and channel weights corresponding to each of the first feature maps, the second feature map corresponding to each of the first feature maps is determined.

[0094] Optionally, the input module 920 is specifically used for: Perform bilinear interpolation on all non-first second feature maps to obtain third feature maps corresponding to each non-first second feature map; each third feature map has the same resolution as the first second feature map; The first second feature map and all third feature maps are concatenated according to the channel dimension to obtain the concatenated feature map.

[0095] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a salient target detection and depth prediction method. This method includes: acquiring an underwater image of a target; inputting the underwater image of the target into an image detection and prediction model to obtain a salient image of the target and a depth image of the target corresponding to the underwater image; the image detection and prediction model is used to align the features of the underwater image of the target with coordinates in a preset coordinate map, and decode the aligned feature map based on a neighborhood integration strategy to obtain the salient image of the target and the depth image of the target; the aligned feature map has the same spatial resolution as the preset coordinate map.

[0096] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the salient target detection and depth prediction methods provided by the above methods. The method includes: acquiring a target underwater image; inputting the target underwater image into an image detection and prediction model to obtain a target salient image and a target depth image corresponding to the target underwater image; the image detection and prediction model is used to align the features of the target underwater image with coordinates in a preset coordinate map, and decode the aligned feature map based on a neighborhood integration strategy to obtain the target salient image and the target depth image; the aligned feature map has the same spatial resolution as the preset coordinate map.

[0098] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the salient target detection and depth prediction methods provided by the above methods. The method includes: acquiring an underwater image of a target; inputting the underwater image of the target into an image detection and prediction model to obtain a target salient image and a target depth image corresponding to the underwater image of the target; the image detection and prediction model is used to align the features of the underwater image of the target with coordinates in a preset coordinate map, and to decode the aligned feature map based on a neighborhood integration strategy to obtain the target salient image and the target depth image; the aligned feature map has the same spatial resolution as the preset coordinate map.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A salient object detection and depth prediction method, characterized in that, The method comprises: obtaining a target underwater image; inputting the target underwater image into an image detection prediction model to obtain a target saliency image and a target depth image corresponding to the target underwater image; the image detection prediction model is used for aligning features of the target underwater image with coordinates in a preset coordinate map, and decoding the aligned feature map based on a neighborhood integration strategy to obtain the target saliency image and the target depth image; the spatial resolution of the aligned feature map is the same as that of the preset coordinate map.

2. The salient object detection and depth prediction method of claim 1, wherein, the image detection prediction model comprises a multi-scale feature extraction unit, a feature compression and optimization unit, a feature coordinate alignment unit and a feature decoding unit; the inputting of the target underwater image into the image detection prediction model to obtain the target saliency image and the target depth image corresponding to the target underwater image comprises: inputting the target underwater image into the multi-scale feature extraction unit to obtain at least two first feature maps corresponding to different scales respectively; inputting the at least two first feature maps corresponding to different scales respectively into the feature compression and optimization unit to obtain second feature maps corresponding to the first feature maps respectively; inputting all the second feature maps into the feature coordinate alignment unit to obtain an aligned feature map and a coordinate offset; inputting the aligned feature map and the coordinate offset into the feature decoding unit to obtain the target saliency image and the target depth image corresponding to the target underwater image.

3. The salient object detection and depth prediction method of claim 2, wherein, the inputting of all the second feature maps into the feature coordinate alignment unit to obtain an aligned feature map and a coordinate offset comprises: based on all the second feature maps, a spliced feature map is obtained; determining potential encodings closest to each query coordinate in the preset coordinate map in the spliced feature map, and based on the coordinates where each potential encoding is located and each corresponding query coordinate, determining a coordinate offset corresponding to each potential encoding to align the aligned feature map with the coordinates of the preset coordinate map; based on all the potential encodings, the aligned feature map is determined.

4. The salient object detection and depth prediction method of claim 2, wherein, the inputting of the aligned feature map and the coordinate offset into the feature decoding unit to obtain the target saliency image and the target depth image corresponding to the target underwater image comprises: determining a normalized weight of a neighborhood potential encoding corresponding to each query coordinate in the aligned feature map; based on the aligned feature map, the coordinate offset and each normalized weight, a target saliency value and a target depth value corresponding to each query coordinate are respectively obtained; based on the target saliency values corresponding to all the query coordinates, the target saliency image is determined; based on the target depth values corresponding to all the query coordinates, the target depth image is determined.

5. The salient object detection and depth prediction method of claim 4, wherein, the determination of the normalized weight of the neighborhood potential encoding corresponding to each query coordinate in the aligned feature map comprises: determining a diagonal rectangular area corresponding to each neighborhood potential encoding and a total rectangular area formed by all the neighborhood potential encodings; the diagonal rectangular area is a rectangular area formed by a diagonal potential encoding in a diagonal line direction of the neighborhood potential encoding and the query coordinate. Based on each of the diagonal rectangular area and each of the total rectangular area, a normalized weight corresponding to each of the neighborhood potential encoding is determined.

6. The salient object detection and depth prediction method of claim 2, wherein, The inputting the first feature map corresponding to each of the at least two scales into the feature compression and optimization unit to obtain a second feature map corresponding to each of the first feature maps comprises: performing convolution operation on each of the first feature maps to obtain a compressed feature map corresponding to each of the first feature maps; determining a channel weight corresponding to each of the first feature maps based on the channel attention mechanism; determining a second feature map corresponding to each of the first feature maps based on the compressed feature map and the channel weight corresponding to each of the first feature maps.

7. The salient object detection and depth prediction method of claim 3, wherein, The obtaining the spliced feature map based on all the second feature maps comprises: performing bilinear interpolation operation on all the non-first second feature maps to obtain a third feature map corresponding to each of the non-first second feature maps; each of the third feature maps has the same resolution as the first second feature map; splicing the first second feature map and all the third feature maps in the channel dimension to obtain the spliced feature map.

8. A salient object detection and depth prediction apparatus characterized by comprising: It comprises: an acquisition module configured to acquire a target underwater image; an input module configured to input the target underwater image into an image detection prediction model to obtain a target saliency image and a target depth image corresponding to the target underwater image; the image detection prediction model is configured to align features of the target underwater image with coordinates in a preset coordinate map, and decode an aligned feature map after the alignment based on a neighborhood integration strategy to obtain the target saliency image and the target depth image; the aligned feature map has the same spatial resolution as the preset coordinate map.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the saliency target detection and depth prediction method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the saliency target detection and depth prediction method according to any one of claims 1 to 7.

Images