Video salient object detection model training method and device, electronic equipment and storage medium

Abstract

This application discloses a training method, apparatus, electronic device, and computer-readable storage medium for a video salient object detection model. Addressing the issues of insufficient multimodal fusion, temporal instability, and high label dependence, a two-stage training framework is proposed: The first stage uses cross-modal unsupervised contrastive learning to mine consistency and complementarity information between RGB and deep modalities, generating and iteratively optimizing salient object pseudo-labels to improve their quality; the second stage uses the optimized salient object pseudo-labels as supervision signals, selecting historical frames and adjacent frames to construct a reference set, training the target model through temporal feature fusion, and iteratively updating network parameters, enabling the model to obtain a stable representation in the time dimension, enhancing its ability to model long-term and short-term dependencies, suppressing dynamic interference, and maintaining target continuity. This method eliminates the need for manually labeled data, effectively reducing data costs through cross-modal contrastive learning and temporal feature fusion, while improving detection accuracy and robustness in complex scenes.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a training method for a video salient target detection model, a training device for a video salient target detection model, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Video salient object detection (SOD) aims to automatically identify the most visually attractive targets in images or videos, and has significant application value in fields such as robot dynamic scene understanding, autonomous driving, and video editing. Compared to static image scenes, video salient object detection not only needs to extract spatially salient cues from a single frame, but also needs to maintain a continuous characterization of the target in the temporal dimension to cope with changes in target appearance, motion, occlusion, and dynamic background interference.

[0003] To enhance perception capabilities in complex scenes, researchers have further introduced multi-source perceptual information. For example, in multimodal perception, RGB images provide semantic and textural cues, while depth / structural information supplements spatial geometric cues, exhibiting a degree of complementarity. However, in practical applications, multimodal data differ in their representational forms and noise characteristics, and simple fusion can easily lead to inconsistencies or mutual interference. Furthermore, temporal fluctuations in dynamic video scenes amplify these instabilities. On the other hand, a significant portion of existing methods rely on large amounts of pixel-level labeled data for training, resulting in high data labeling costs and long cycles, and limitations in adaptability to diverse real-world scenes. Therefore, how to effectively utilize multimodal and temporal information while reducing labeling dependence, thereby improving the accuracy and robustness of salient object detection in complex dynamic video scenes, remains a subject for further research. Summary of the Invention

[0004] This application provides a training method, a training device, an electronic device, and a computer-readable storage medium for a video salient object detection model. Through a two-stage collaborative optimization training mechanism, the first stage utilizes cross-modal unsupervised contrastive learning and neighborhood optimization to generate and optimize the quality of salient object pseudo-labels, reducing reliance on manual annotation while improving multimodal fusion effects and the quality of salient object pseudo-labels. The second stage trains the model and optimizes parameters through reference set construction and temporal feature fusion, enabling the model to obtain stable representations in the time dimension and enhancing its ability to model long-term and short-term dependencies, thereby improving the accuracy and robustness of video salient object detection.

[0005] Firstly, this application provides a training method for a video salient object detection model, the training method including: Acquire RGB-D video sequences captured by the robot while it is performing a preset task to obtain RGB-D data for each frame; each frame of RGB-D data includes an RGB image and a corresponding depth image; Each frame of RGB image and the corresponding frame of depth image are input into the feature extraction network to obtain each RGB feature and each depth feature. A cross-modal contrastive learning loss is constructed based on each RGB feature and each depth feature. The network parameters of the feature extraction network are iteratively updated according to the cross-modal contrastive learning loss until the first preset training condition is met. After meeting the first preset training conditions, pseudo-labels for each video salient target are generated based on the updated network parameters and the RGB-D data of each frame are updated iteratively within a preset neighborhood based on each RGB feature, each depth feature and the corresponding pixel spatial neighborhood relationship. The video salient target detection model is trained using the iteratively updated pseudo-labels of each video salient target as the supervision signal. A reference set is constructed based on the RGB-D data of the current frame, the RGB-D data of its neighboring frames, and historical reference frames. Temporal features are fused through the reference set to train the video salient target detection model until the second preset training condition is met, thus obtaining the trained video salient target detection model.

[0006] Furthermore, a cross-modal contrastive learning loss is constructed based on each RGB feature and each deep feature, including: Activation operations are performed on each RGB feature map and each depth feature map to obtain each RGB activation map and each depth activation map; The intra-sample contrast significance loss and inter-sample contrast significance loss are calculated based on each RGB feature map, each depth feature map, each RGB activation map, and each depth feature map. The modal contrast learning loss is determined based on the intra-sample contrast significance loss and the inter-sample contrast significance loss.

[0007] Further, based on each RGB feature map, each depth feature map, each RGB activation map, and each depth feature map, the intra-sample contrast significance loss and the inter-sample contrast significance loss are calculated, including: For each frame of RGB-D data, RGB foreground and background representations are determined based on the corresponding RGB feature map and RGB activation map; depth foreground and depth background representations are determined based on the corresponding depth feature map and depth activation map; and foreground activation is calculated based on the RGB foreground and depth foreground representations. Figure 1 Consistency; Calculating background activation based on RGB background representation and depth background representation Figure 1 To the point of being compatible; Based on each foreground activation Figure 1 Consistency determines the positive sample loss of the prospect; Activation based on various backgrounds Figure 1 Consistency determination of background positive sample loss; The in-sample contrast significance loss is determined based on the foreground positive sample loss and the background positive sample loss; The intermodal negative sample loss is determined based on the RGB foreground representation and the background representation at each depth. Intramodal negative sample loss is determined based on each RGB foreground representation and each RGB background representation; The significance loss between samples is determined based on the negative sample loss between modalities and the negative sample loss within modalities.

[0008] Furthermore, based on each foreground activation Figure 1 Consistency-based positive foreground sample loss includes; Activate based on each foreground Figure 1 Homogeneity is activated in various foregrounds Figure 1 The ranking in consistency determines the corresponding dynamic weights of the foreground; based on the activation of each foreground... Figure 1 Consistency and corresponding dynamic weights are used to calculate the loss of positive prospect samples; Activation based on various backgrounds Figure 1 Consistency determination of background positive sample loss includes: Activate based on each background Figure 1 Activation of homogeneity in various backgrounds Figure 1 Ranking in consistency determines background dynamic weights; activation based on each background Figure 1 Consistency and corresponding background dynamic weights are used to calculate the background positive sample loss.

[0009] Furthermore, the network parameters of the feature extraction network are iteratively updated based on the cross-modal contrastive learning loss until the first preset training condition is met, including: The total target loss is determined based on cross-modal contrastive learning loss, dynamic local binary cross-entropy, and gated conditional random field loss. The network parameters of the feature extraction network are iteratively updated based on the target total loss until the target total loss is less than the first loss threshold.

[0010] Furthermore, within a preset neighborhood, the pseudo-labels for each salient target in the video are iteratively updated based on each RGB feature, each depth feature, and the corresponding pixel spatial neighborhood relationship, including: Based on a pre-built multimodal online optimizer, the pseudo-labels for each video's salient targets are iteratively updated: For each pixel in the pseudo-label of a salient target in the video, calculate the RGB feature distance, depth feature distance, and spatial location distance between the pixel and its neighboring pixels; The saliency responses of neighboring pixels are weighted and fused based on RGB feature distance, depth feature distance, and spatial location distance to generate an updated saliency response; The pseudo-labels of salient targets in the video are iteratively updated based on the updated salient response until the preset convergence condition is met.

[0011] Furthermore, the online optimizer is defined as:

[0012] The iterative optimization function is:

[0013] in, Represents the set of 8 neighboring pixels. and The weighting parameters are used to balance the importance of features. , and These are the RGB feature distance, depth feature distance, and positional distance, respectively. , and The corresponding standard deviation, Denotes the Hadamard product of matrices. For the first The video salient target pseudo-labels are updated in the next iteration. For the first The video salient target pseudo-labels are updated in the next iteration.

[0014] Furthermore, a reference set is constructed based on the current frame's RGB-D data, its neighboring frames' RGB-D data, and historical reference frames, including: For each frame of historical RGB-D data within a preset time period, calculate the quality score of the structural similarity between the pseudo-labels of significant targets corresponding to each frame of historical RGB-D data and the current frame of RGB-D data. Select at least one historical RGB-D data frame with the highest quality score from each frame of historical RGB-D data as the historical reference frame RGB-D data for the current frame RGB-D data; A reference set is formed by combining the RGB-D data of historical reference frames with a specified number of adjacent frames corresponding to the RGB-D data of the current frame.

[0015] Secondly, this application provides a salient target detection method for use in robots. The salient target detection method includes: Acquire the target RGB-D video sequence captured by the robot during the current task; Input the target RGB-D video sequence into the video salient object detection model to obtain the salient object detection results; Based on the results of salient target detection and the robot's current task status, control the robot's operation; The video salient object detection model is obtained after training using any of the training methods in the first aspect.

[0016] Thirdly, this application provides a training device for a video salient object detection model, comprising: an acquisition module for acquiring RGB-D video sequences collected when a robot performs a preset task, so as to obtain RGB-D data for each frame; each frame of RGB-D data includes an RGB image and a corresponding depth image; The first-stage training module is used to input each frame of RGB image and the corresponding frame of depth image into the feature extraction network to obtain each RGB feature and each depth feature, and to construct a cross-modal contrastive learning loss based on each RGB feature and each depth feature. The network parameters of the feature extraction network are iteratively updated according to the cross-modal contrastive learning loss until the first preset training condition is met. The optimization module is used to generate pseudo-labels for each video salient target corresponding to each frame of RGB-D data based on the updated network parameters after the first preset training conditions are met, and to iteratively update each video salient target pseudo-label based on each RGB feature, each depth feature and the corresponding pixel spatial neighborhood relationship within a preset neighborhood. The two-stage training module is used to train the video salient target detection model with the updated pseudo-labels of each video salient target as the supervision signal. It constructs a reference set based on the RGB-D data of the current frame, the RGB-D data of its neighboring frames, and historical reference frames, and performs temporal feature fusion through the reference set to train the video salient target detection model until the second preset training condition is met to obtain the trained video salient target detection model.

[0017] Fourthly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the training methods described in the first aspect.

[0018] Fifthly, this application provides a robot including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the second aspect.

[0019] In a sixth aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first or second aspect.

[0020] In a seventh aspect, this application provides a computer program product comprising a computer program that, when executed by one or more processors, implements the steps of the method as described in the first or second aspect.

[0021] Compared with existing technologies, the beneficial effects of this application are as follows: This application addresses the problems of insufficient multimodal fusion effect, insufficient utilization of temporal information, and strong dependence on large-scale pixel-level annotation in existing video salient object detection methods in complex dynamic scenes. It proposes a phased model training mechanism, which improves model performance and training efficiency in a coordinated manner from three levels: representation learning, supervision signal construction, and temporal modeling.

[0022] In the first stage, a cross-modal contrastive learning mechanism based on RGB and deep features is constructed to align and coordinate the representations of different modalities without manual annotation. This enables the semantic texture information of the RGB modality and the spatial structural information of the deep modality to form a consistent and complementary expression in a unified feature space, effectively mitigating the problems of distribution differences and noise interference between modalities and significantly improving the effectiveness and stability of multimodal fusion. Based on this, pseudo-labels for salient targets in the video are generated using the fused high-consistency features, and iterative optimization is performed by combining pixel spatial neighborhood relationships. This ensures that the pseudo-labels for salient targets satisfy structural continuity and boundary consistency constraints within local regions, thereby refining the target contours while suppressing noise errors. Overall, this improves the accuracy, smoothness, and reliability of the pseudo-labels for salient targets, providing high-quality supervision signals for subsequent training.

[0023] In the second stage, high-quality salient target pseudo-labels optimized by neighborhood are used as supervision signals. A temporal modeling strategy oriented towards video sequences is introduced. By forming a reference set with historical reference frames and recent neighboring frames, long-term memory and short-term memory are fused under a unified framework to explicitly model long-range and short-range temporal dependencies. Furthermore, the target model is trained using temporal fusion features and the network parameters are continuously iteratively updated. This enables the model to collaboratively utilize cross-frame contextual information in the temporal dimension, improving its ability to capture spatiotemporal dependencies. This better addresses complex situations such as changes in target appearance, rapid movement, and occlusion, effectively enhancing the temporal consistency and discriminative ability of feature representation, and ensuring stable detection and coherent representation of salient targets in consecutive frames of the video.

[0024] In summary, this application achieves effective learning under unlabeled conditions through two-stage collaborative optimization. On the one hand, it significantly improves the multimodal fusion effect and the quality of salient target pseudo-labels by utilizing cross-modal unsupervised contrastive learning and neighborhood constraint mechanisms. On the other hand, it promotes the full utilization of long-term and short-term temporal information and drives model parameter optimization by constructing reference sets and fusing temporal features. This results in higher accuracy, stronger robustness, and better temporal consistency in video salient target detection in complex dynamic scenes.

[0025] It is understandable that the beneficial effects of aspects two through five can be found in the relevant descriptions in aspect one, and will not be repeated here. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the training method of the video salient object detection model provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the salient target detection model provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the training device for the video salient object detection model provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0029] Video salient object detection aims to identify the most visually attractive targets in images or videos, and has significant application value in fields such as robot dynamic scene understanding, autonomous driving, and video editing. Compared to static images, video salient object detection (VSOD) not only needs to focus on the spatial salient information of a single frame, but also needs to continuously track the changes of the target in the temporal dimension to ensure the continuity and stability of the detection results. To adapt to complex dynamic scenes, researchers have introduced multimodal data such as depth maps and proposed an RGB-D video salient object detection method. By fusing the semantic and texture information of RGB images with the spatial structure and distance information of depth images, the detection performance in complex environments has been improved to a certain extent.

[0030] However, most existing RGB-D video salient object detection methods rely on fully supervised training paradigms, which are heavily dependent on large-scale pixel-level labeled data, resulting in high annotation costs and difficulty in adapting to diverse real-world scenarios. Although unsupervised video salient object detection (USOD) has made some progress, existing methods are mainly designed for single-modal static images, making it difficult to fully utilize the complementary information between RGB and depth modalities. Furthermore, they lack sufficient modeling of temporal dependencies in video sequences, thus limiting their applicability and performance in RGB-D video scenarios.

[0031] Specifically, on the one hand, RGB and depth modalities differ significantly in their representation and statistical characteristics. Existing methods often employ simple fusion strategies, failing to effectively mine the consistency and contrast information between modalities, resulting in insufficient feature representation capabilities. On the other hand, the long-range and short-range temporal dependencies in video data are not fully utilized, making it difficult to guarantee the consistency and stability of salient targets in consecutive frames. Furthermore, in unsupervised frameworks, the quality of salient target pseudo-labels directly affects model training performance, while existing methods lack effective optimization mechanisms for multimodal information, resulting in generated salient target pseudo-labels with high noise and unstable structure, thus limiting the improvement of overall detection performance.

[0032] To address the aforementioned issues, this application proposes a training method for a video salient object detection model. The framework is based on a two-stage collaborative optimization mechanism. Specifically, the first stage utilizes cross-modal unsupervised contrastive learning and neighborhood optimization to generate and optimize salient object pseudo-labels, reducing reliance on manual annotation while improving multimodal fusion performance and the quality of salient object pseudo-labels. The second stage trains the model through reference set construction and temporal feature fusion to optimize parameters, enabling the model to achieve stable representations over time and enhancing its ability to model long- and short-term dependencies, thereby improving the accuracy and robustness of video salient object detection. The training method proposed in this application will be illustrated below with specific embodiments.

[0033] The training method for the video salient object detection model provided in this application embodiment can be applied to electronic devices such as mobile phones, tablets, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). It can also be deployed on edge computing nodes or carried out by robots as the execution subject for data collection and model training. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0034] It should be noted that although training based on robots can directly obtain multimodal data that closely resembles real-world task scenarios, it still has certain limitations in practical applications. For example, there are issues such as high hardware costs, data acquisition efficiency being constrained by the physical environment, limited scene coverage, and the impact of equipment operational stability on data quality. These issues limit the scale and efficiency of training to some extent. Therefore, it is preferable to use electronic devices with powerful computing capabilities for model training.

[0035] To illustrate the technical solutions proposed in this application, the following description will use an electronic device as the execution subject to illustrate various embodiments.

[0036] Figure 1 A schematic flowchart illustrating the training method of the video salient object detection model provided in this application is shown. The training method of the video salient object detection model includes: Step 110: The electronic device acquires the RGB-D video sequence collected when the robot performs the preset task to obtain RGB-D data for each frame.

[0037] When the robot performs preset tasks such as navigation, autonomous driving, or video editing, it can acquire RGB-D video sequences in real time through its integrated depth sensor, or receive RGB-D video sequences uploaded by users, and detect salient targets in the video to assist in environmental perception and / or decision-making. Each frame of RGB-D data consists of an RGB image and a depth image aligned with it in time and space. The RGB image mainly provides color, texture, and semantic information, such as the edge contours of target objects, color contrast, and semantic region distribution; the depth image provides pixel-level distance information and spatial structure constraints, such as the depth difference between the foreground and background and the geometric shape of objects, thus providing supplementary support for the accurate localization and segmentation of salient targets in the video.

[0038] Therefore, multimodal observation data with good temporal continuity can be obtained. The multimodal data collected directly from real dynamic scenes can simultaneously possess spatial structure information and temporal continuity characteristics, thus providing reliable data support for models to characterize salient video targets in complex environments.

[0039] Step 120: The electronic device inputs each frame of RGB image and the corresponding frame of depth image into the feature extraction network to obtain each RGB feature and each depth feature, and constructs a cross-modal contrastive learning loss based on each RGB feature and each depth feature. The network parameters of the feature extraction network are iteratively updated according to the cross-modal contrastive learning loss until the first preset training condition is met.

[0040] Step 130: After meeting the first preset training conditions, the electronic device generates pseudo-labels for each video salient target corresponding to each frame of RGB-D data based on the updated network parameters, and iteratively updates each video salient target pseudo-label within a preset neighborhood based on each RGB feature, each depth feature and the corresponding pixel spatial neighborhood relationship.

[0041] To achieve unsupervised training, the training method for the video salient object detection model employs a two-stage training strategy. The advantage of this strategy lies in decoupling "representation learning" from "object modeling": the first stage focuses on training the feature extraction network, learning robust feature representations through cross-modal constraints without manual annotation, and generating initial salient object pseudo-labels accordingly. Subsequently, spatial neighborhood consistency is used to iteratively optimize the salient object pseudo-labels, thereby gradually improving their accuracy and structural rationality, providing high-quality supervision signals for subsequent training. The first stage of training specifically includes: The electronic device inputs the RGB images and corresponding depth images of each frame into a feature extraction network (e.g., a parallel coding structure or a dual-branch network) to extract RGB and depth features. For example, the RGB branch can extract the texture boundary features of the object, while the depth branch can extract the structural features of the foreground convex region. Based on this, a cross-modal contrastive learning loss is constructed. For example, features of the same pixel or the same region in both RGB and depth modalities are used as positive sample pairs, and features from different spatial locations or different frames are used as negative sample pairs. By using contrast constraints to bring positive sample features closer together and widen the distance between negative sample features, the alignment and differentiation of multimodal features in the semantic space are achieved. Subsequently, the feature extraction network is iteratively updated based on this loss until the first preset training condition is met.

[0042] The first preset training condition is used to determine whether the feature extraction network based on cross-modal contrastive learning in the first stage has completed effective training and reached the expected convergence state, thus serving as the basis for deciding whether to terminate the current stage of training and proceed to the next stage. This condition is essentially a constraint on the sufficiency and stability of model training, and can be set based on indicators such as loss convergence, feature consistency, or training epochs.

[0043] For example, in a concrete implementation, the convergence criterion can be that the change in cross-modal contrastive learning loss over several consecutive iterations is less than a preset threshold (e.g., less than 0.001); or, the condition can be satisfied when the similarity index (e.g., cosine similarity) between RGB features and deep features reaches a preset level, indicating that the two modalities have achieved good alignment in the feature space; a maximum number of training epochs (e.g., reaching a preset number of epochs) can also be used as a supplementary constraint to avoid excessively long training processes or overfitting. Training based on this first preset training condition ensures that the feature extraction network has a stable and discriminative multimodal feature representation capability before entering the stage of generating salient target pseudo-labels.

[0044] After training, even without manual annotation, electronic devices can achieve multimodal feature alignment through the trained feature extraction network, fully exploring the consistency and complementarity of RGB and depth information to generate corresponding pseudo-labels for salient video targets.

[0045] To further improve the quality of salient target pseudo-labels, within a preset neighborhood (such as a local window centered on a pixel), the electronic device can iteratively optimize the salient target pseudo-labels by combining the boundary information of RGB features with the structural consistency of depth features. For example, for boundary regions, if the RGB features show obvious edges causing abrupt changes in depth, then this region is strengthened as the boundary of a salient target in the video; for noisy regions, smoothing is performed through neighborhood consistency constraints. Through multiple iterations, the salient target pseudo-labels gradually approach a state of clear boundaries and consistent regions, significantly improving the accuracy and structural rationality of salient target pseudo-labels without manual annotation, providing reliable signals for subsequent supervised training.

[0046] Step 140: The electronic device trains the video salient target detection model using the iteratively updated pseudo-labels of each video salient target as the supervision signal. A reference set is constructed based on the RGB-D data of the current frame and its neighboring frames and historical reference frames. Temporal features are fused through the reference set to train the video salient target detection model until the second preset training condition is met to obtain the trained video salient target detection model.

[0047] After obtaining optimized pseudo-labels for salient video targets, these are used as supervisory signals in the second stage of training. This stage uses video sequences as the modeling object. A reference set is constructed, consisting of the RGB-D data of the best historical reference frame and the RGB-D data of the adjacent frames corresponding to the current frame's RGB-D data. Long-term and short-term temporal information are fused within a unified framework to obtain temporal fusion features. These features are then used to train the target model and iteratively update the network parameters, enabling the model to learn stable and discriminative feature representations over time, thereby improving its ability to continuously characterize salient video targets in dynamic scenes. The second stage of training specifically includes: For each current frame's RGB-D data, a reference set is constructed by selecting RGB-D data from its neighboring frames (representing short-term dynamic information) and RGB-D data from historical reference frames (representing long-term stability information). Neighboring frames characterize the target's short-term motion changes and local temporal continuity, while historical reference frames provide information on the target's appearance and structure at times with no occlusion or minimal interference. Temporal feature fusion is then performed based on this reference set. In this process, the electronic device receives the feature map of the current frame and retrieves corresponding historical reference frame features from the "feature memory." Through attention mechanisms and feature concatenation operations, long-term and short-term memory information is incorporated into the feature representation of the current frame, resulting in fused features with temporal context constraints. Subsequently, the fused enhanced features are input into the decoder, and the video salient target detection model is trained and its network parameters iteratively updated accordingly. This allows the model to effectively "backtrack" to historical saliency information when predicting the current frame, synergistically utilizing long- and short-term temporal dependencies to improve its modeling ability for dynamic changes, occlusion, and complex backgrounds, thereby enhancing the temporal consistency and overall robustness of the detection results. By continuously training until the second preset training condition is met, the trained video salient object detection model can be obtained.

[0048] The second preset training condition is used to determine whether the video salient object detection model based on salient object pseudo-label supervision and temporal feature fusion in the second stage has completed sufficient training and achieved stable performance, thus serving as the basis for terminating model training and outputting the final model. This condition focuses on the model's convergence and generalization ability on the video salient object detection task, and is usually set based on detection performance indicators, loss function convergence, or training epochs.

[0049] For example, in a concrete implementation, the convergence criterion can be that the improvement of the model's performance metrics (such as F-measure, MAE, or IoU) on the validation set is less than a preset threshold (such as less than 0.001) over several consecutive training rounds; or, the condition can be determined to be met when the overall loss function (such as cross-entropy loss based on salient target pseudo-labels or structural similarity loss) tends to stabilize after multiple iterations; a maximum number of training rounds (such as reaching a preset number of epochs) can also be set as a supplementary termination condition to control training costs and avoid overfitting.

[0050] Both preset training conditions are formally training termination criteria, and can be set based on loss convergence, performance metrics, or training epochs. Their purpose is to ensure the model reaches a stable state before proceeding to the next stage or ending training. However, their focuses differ: the first preset training condition applies to the cross-modal contrastive learning stage, emphasizing the representation learning ability of the feature extraction network and primarily measuring whether multimodal features achieve effective alignment and discrimination; while the second preset training condition applies to the supervised training stage of the video salient object detection model, focusing more on the model's performance on specific detection tasks and its temporal modeling effectiveness, emphasizing detection accuracy, temporal consistency, and overall robustness. Therefore, the two conditions are clearly distinguishable in their training stages and evaluation objectives, while simultaneously forming a sequential and progressively converging relationship within the overall training process.

[0051] In this embodiment, the electronic device first acquires a multimodal data foundation with spatial structure and temporal continuity using RGB-D video sequences collected in real-world scenarios. Secondly, it achieves effective alignment and complementary fusion of RGB and depth features through cross-modal unsupervised contrastive learning, obtaining highly discriminative salient feature representations without manual annotation. Furthermore, it significantly improves the accuracy and structural consistency of salient target pseudo-labels through salient target pseudo-label generation and pixel-neighborhood-based iterative optimization, thereby constructing a high-quality supervisory signal. Finally, it constructs a reference set and performs temporal feature fusion by filtering historical reference frames and adjacent frames, training and optimizing the model's parameters. This enables the model to fully utilize long-term and short-term temporal dependencies, enhancing its modeling ability for dynamic changes and complex scenes, and comprehensively improving the accuracy, temporal consistency, and robustness of video salient target detection while reducing reliance on manual annotation.

[0052] In some embodiments, to improve the ability of the feature extraction network to represent and distinguish salient targets in a video, the electronic device constructs a cross-modal contrastive learning loss based on each RGB feature and each depth feature, including: Step A1: The electronic device performs activation operations on each RGB feature map and each depth feature map to obtain each RGB activation map and each depth activation map.

[0053] The electronic device performs activation operations on each RGB feature map and each depth feature map to highlight response regions related to salient targets in the video. These activation operations can be implemented through normalization, thresholding, or attention weighting to enhance locations in the original features that strongly respond to salient targets while suppressing background noise. For example, in the RGB feature map, regions with significant color contrast or edge structures will exhibit higher responses after activation; in the depth feature map, regions with significant depth differences between the foreground and background will be enhanced, resulting in RGB activation maps and depth activation maps that provide explicit references for subsequent saliency constraints.

[0054] Step A2: The electronic device calculates the intra-sample contrast significance loss and the inter-sample contrast significance loss based on each RGB feature map, each depth feature map, each RGB activation map, and each depth feature map.

[0055] Intra-sample contrastive saliency loss is used to constrain the consistency of salient regions between different modalities within the same frame. For example, it treats high-response regions in the RGB activation map and their corresponding regions in the depth activation map as positive sample pairs, while treating salient and non-salient regions as negative sample pairs. By narrowing the feature distance between positive samples and widening the feature distance between negative samples, it achieves saliency alignment between modalities. For instance, for the same object region, regions prominent in RGB due to color and regions prominent in depth due to distance should remain consistent in the feature space, while background regions should be distinguished. Inter-sample contrastive saliency loss is used to enhance the discriminative power between different samples. For example, it treats salient target regions in videos from different frames or scenes as distinguishable samples, using contrast constraints to prevent the model from misclassifying different targets or backgrounds as the same salient region. This loss helps improve the model's ability to distinguish diverse salient targets in complex scenes.

[0056] In order to fully explore the consistency and complementary information between RGB and depth modalities, electronic devices can be based on RGB feature maps, depth feature maps and their corresponding activation maps, as well as intra-sample contrast significance loss and inter-sample contrast significance loss.

[0057] Step A3: The electronic device determines the modal contrast learning loss based on the intra-sample contrast significance loss and the inter-sample contrast significance loss.

[0058] After obtaining the two types of losses, the electronic device can calculate the total loss based on the intra-sample contrast significance loss and the inter-sample contrast significance loss, for example, by directly adding them or by performing weighted fusion, to obtain the modality contrastive learning loss, and optimize the feature extraction network accordingly. By simultaneously considering intra-sample consistency and inter-sample differences, the feature extraction network maintains good discriminative ability while learning multimodal alignment, thereby obtaining a more robust feature representation.

[0059] In this embodiment, the electronic device optimizes the feature extraction network by using modal contrastive learning loss calculated based on intra-sample contrastive saliency loss and inter-sample contrastive saliency loss. On the one hand, it can highlight salient regions and suppress background interference. On the other hand, it can achieve cross-modal saliency alignment within the same sample and enhance feature discrimination between different samples. This improves the discriminative and generalization capabilities of multimodal features under unsupervised conditions, providing a high-quality feature foundation for subsequent generation of salient target pseudo-labels and model training.

[0060] In some embodiments, the electronic device calculates intra-sample contrast significance loss and inter-sample contrast significance loss based on each RGB feature map, each depth feature map, each RGB activation map, and each depth feature map, including: Step A21: For each frame of RGB-D data, the electronic device determines the RGB foreground and RGB background representations based on the corresponding RGB feature map and the corresponding RGB activation map; it also determines the depth foreground and depth background representations based on the corresponding depth feature map and the corresponding depth activation map; and it calculates the foreground activation based on the RGB foreground and depth foreground representations. Figure 1 Consistency; Calculating background activation based on RGB background representation and depth background representation Figure 1 To the point of being responsive.

[0061] For each frame of RGB-D data, RGB foreground and background representations are separated by combining the RGB feature map and its corresponding RGB activation map. Simultaneously, depth foreground and background representations are extracted based on the depth feature map and its corresponding depth activation map. Furthermore, the consistency between the RGB and depth foreground representations can be measured to characterize the consistency of the foreground region's response across different modalities. For example, a salient object in the same video might appear as a high-contrast edge in RGB and as a convex structure in depth, indicating a high degree of similarity in their features. Similarly, by measuring the consistency between the RGB and depth background representations, the uniformity of the background region across different modalities is characterized, thus providing a basis for distinguishing between salient and non-salient regions.

[0062] Among them, foreground / background activation Figure 1Consistency can be calculated using similarity measures between features. For example, for foreground consistency, the RGB foreground representation and the depth foreground representation can be normalized, and then cosine similarity can be used to calculate the angle similarity between them in the feature space. When both have consistent response patterns in the salient target region of the video (such as simultaneously highlighting the boundary or structure of the same object), the similarity is high, thus representing strong consistency. Similarly, background consistency can be obtained by calculating the cosine similarity or the inverse Euclidean distance between the RGB background representation and the depth background representation, which is used to characterize the distribution consistency of non-salient regions under different modalities. In this way, the consistency of cross-modal features can be quantitatively evaluated from both the foreground and background levels.

[0063] Step A22: The electronic device is activated based on each foreground element. Figure 1 Consistency determines the positive sample loss for the prospect.

[0064] Step A23: Electronic devices are activated based on various backgrounds. Figure 1 Consistency determines the background positive sample loss.

[0065] Step A24: The electronic device determines the intra-sample contrast significance loss based on the foreground positive sample loss and the background positive sample loss.

[0066] Based on the aforementioned consistency metrics, foreground positive sample loss and background positive sample loss are constructed respectively. The foreground positive sample loss is used to constrain the aggregation degree of salient region features under different modalities, ensuring that the same target maintains a compact distribution across modal spaces. The background positive sample loss is used to constrain the consistency of background regions, avoiding background noise shifting between different modalities. On this basis, the positive sample losses of the foreground and background are fused to obtain the intra-sample contrast saliency loss, thereby achieving the dual constraints of salient region alignment and background suppression within a single sample.

[0067] Step A25: The electronic device determines the intermodal negative sample loss based on the RGB foreground representation and the background representation at each depth.

[0068] Step A26: The electronic device determines the intramodal negative sample loss based on each RGB foreground representation and each RGB background representation.

[0069] Step A27: The electronic device determines the significance loss of inter-modal negative sample loss based on the inter-modal negative sample loss and the intra-modal negative sample loss.

[0070] Meanwhile, feature discrimination capabilities are further enhanced from cross-sample and cross-regional perspectives. An intermodal negative sample loss is constructed by comparing RGB foreground representations with deep background representations, used to widen the feature distance between salient targets and non-salient regions in different modalities. Simultaneously, an intramodal negative sample loss is constructed by comparing RGB foreground representations with RGB background representations, used to strengthen the distinction between foreground and background within the same modality. Based on this, the two types of negative sample losses are fused to obtain an inter-sample contrast saliency loss, thereby improving the separability of the feature space from a more global perspective.

[0071] In this embodiment, on the one hand, consistency constraints between foreground and background across modalities are achieved within a single sample, making salient regions more concentrated and background regions more stable; on the other hand, effective contrast relationships are constructed between different samples and different regions to enhance the discriminative and distinguishing capabilities of the features. By constructing these two losses, this training method can simultaneously consider feature alignment and feature separation under unsupervised conditions, thereby effectively improving the quality of multimodal feature representation and providing a more reliable and discriminative feature foundation for the subsequent training of video salient object detection models.

[0072] In some embodiments, to accurately calculate the foreground positive sample loss, the electronic device may perform calculations based on each foreground activation. Figure 1 Homogeneity is activated in various foregrounds Figure 1 The ranking in consistency determines the corresponding dynamic weights of the foreground; based on the activation of each foreground... Figure 1 Consistency and corresponding dynamic weights are used to calculate the positive sample loss for the foreground.

[0073] Foreground activation between different sample pairs Figure 1 After achieving consistency, all consistency scores can be sorted to determine the relative position of each foreground sample pair in the overall distribution. Then, a corresponding dynamic foreground weight is assigned to each sample pair based on the sorting results. For example, sample pairs with higher consistency and higher ranking can be assigned greater weights, while sample pairs with lower consistency can have relatively lower weights; conversely, a strategy can be used to give higher weights to sample pairs with lower consistency and lower ranking. Based on this, each foreground is activated. Figure 1 Consistency and corresponding dynamic weights are weighted and fused to calculate the foreground positive sample loss.

[0074] When employing a mechanism that assigns weights in a consistent or positive order, it can adaptively highlight high-quality foreground matching relationships compared to equal-weighting. On one hand, regions with high consistency across modalities and capable of stably representing salient video targets are given greater weight, allowing them to dominate the optimization process and thus strengthening the model's ability to align key salient regions. On the other hand, sample pairs with low consistency and susceptible to noise or occlusion are given less weight, reducing their interference with the model's update direction and thus improving the stability and robustness of training.

[0075] When employing a weighting strategy incorporating exponential decay (such as exp(-rank)), lower-ranked, less consistent sample pairs receive relatively larger weights. This causes the model to focus more on difficult-to-align foreground regions during training, thereby enhancing its ability to model complex situations (such as occlusion, deformation, or modal differences). Thus, through a ranking-based dynamic weighting mechanism, a flexible trade-off can be achieved between "emphasizing reliable samples" and "focusing on difficult samples," further improving the discriminative power of positive foreground sample loss and overall training performance.

[0076] Understandably, the two sorting strategies essentially correspond to two optimization objectives: "emphasizing high-confidence samples" (ascending order) and "focusing on difficult samples" (reverse order). Optionally, the two can be combined to simultaneously ensure stability and discriminability during training.

[0077] For example, the two types of weights can be weighted and fused, such as simultaneously calculating weights based on ascending order and weights based on descending order (or exponential decay), and then obtaining the final dynamic weights through linear combination. This ensures that highly consistent samples dominate the optimization direction in the early stages of training, while gradually introducing low-consistency samples to improve the model's adaptability to complex situations.

[0078] For example, a phased or adaptive strategy can also be adopted. For instance, in the early stages of training, positive weights are mainly used to prioritize learning stable and reliable cross-modal alignment relationships. As training progresses, the proportion of reverse weights is gradually increased so that the model pays more attention to difficult samples, thereby refining the boundaries or handling occluded regions.

[0079] For example, gating or thresholding mechanisms can also be designed to divide samples into high-confidence regions and hard regions based on consistency scores, and different ranking weight strategies can be applied to different regions to achieve more refined dynamic control.

[0080] In other words, by combining strategies, a balance can be achieved between suppressing noise interference and enhancing learning of difficult samples, so that the foreground positive sample loss has both stable convergence and stronger discriminative ability and generalization performance.

[0081] In some embodiments, to accurately calculate the background positive sample loss, the electronic device may activate based on each background sample. Figure 1 Activation of homogeneity in various backgrounds Figure 1 Ranking in consistency determines background dynamic weights; activation based on each background Figure 1 Consistency and corresponding background dynamic weights are used to calculate the background positive sample loss.

[0082] The calculation process for background positive sample loss is basically the same as that for foreground positive sample loss. To improve the accuracy of background positive sample loss calculation, the electronic device obtains background activation values ​​between different sample pairs. Figure 1 After consistency is achieved, all consistency scores can be sorted to determine the relative position of each background sample pair in the overall distribution; subsequently, based on the sorting results, corresponding background dynamic weights are assigned to each sample pair, and the background is activated. Figure 1 The consistency and its corresponding dynamic weights are weighted and fused to obtain the background positive sample loss.

[0083] By introducing a ranking-based dynamic weighting mechanism, adaptive adjustment can be achieved during background region modeling. On the one hand, it can highlight stable background regions with high consistency, enhancing the model's reliable characterization of background distribution. On the other hand, it can also strengthen the constraints on complex or easily confused background regions according to strategy design, thereby improving the distinguishability of background features and the stability of the overall training process. The specific ranking method and dynamic weight allocation strategy have been described in the foregoing embodiments and will not be repeated here.

[0084] In some embodiments, for ease of understanding, the above calculation process can be illustrated using a specific multimodal video salient target scene to enhance understanding and feasibility.

[0085] For example, during a robot's grasping task, a frame of RGB-D data contains a target object (such as a red cup) on a table. In this frame, the RGB feature map primarily highlights the cup's color contrast and edge texture, while the depth feature map highlights the cup's convex structure relative to the table. First, the RGB activation map... With RGB feature map The RGB foreground representation is obtained by fusion and matrix multiplication. Essentially, it uses activation maps to weight feature maps, highlighting features corresponding to salient regions; correspondingly, through and The combination yields the RGB background characterization This is used to characterize non-salient regions. Similarly, it can be based on deep activation maps. With depth feature map Depth foreground representations were obtained respectively. and deep background representation Through this process, the original features can be explicitly decomposed into two semantic subspaces: "foreground" and "background".

[0086] After obtaining the above characterization, the consistency between the foreground and background is further calculated. For example, for the first... Frame and the Frame RGB data, the consistency between RGB foreground representation and depth foreground representation is calculated using cosine similarity. If both accurately depict the cup area, the similarity is high; similarly, the consistency between the RGB background and the depth background... This reflects the degree of matching to the background area (such as the desktop). Based on this, a ranking-based weight is introduced. and More weights are assigned to sample pairs with higher consistency, thereby constructing a positive foreground sample loss. Loss compared to background positive samples Furthermore, the in-sample significance loss was obtained. The purpose of this process is to enhance the consistency of the foreground region across modalities while maintaining stable representation of the background region.

[0087] For example, , , , , , as well as The formula is as follows:

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] in, Represents cosine similarity. To control the hyperparameters of weight smoothness, This represents the number of samples.

[0095] Furthermore, negative sample constraints are constructed from a discriminative perspective. For example, the RGB foreground representation (cup region) is compared with the depth background representation (tabletop region), and the intermodal negative sample loss is calculated. This method aims to differentiate salient targets from non-salient regions in the video across modal spaces; simultaneously, it compares RGB foreground representations with RGB background representations to calculate intramodal negative sample loss. This enhances the foreground-background differentiation within the same modality. The two methods are then combined to obtain the inter-sample contrast significance loss. It is used to improve the discriminativeness of the feature space from a global perspective.

[0096] For example, , as well as The formula is as follows:

[0097]

[0098]

[0099] For different frames, such as in consecutive video frames, the cup may move slightly or be partially occluded. Through the above consistency and contrast constraints, the foreground representation between different frames can still maintain a high degree of similarity, while the background or interference areas are effectively distinguished, thereby enhancing the model's ability to adapt to dynamic changes.

[0100] This example demonstrates that the above computational process achieves explicit decoupling of foreground and background features through activation guidance and strengthens cross-modal alignment using weighted consistency constraints. Furthermore, it enhances the separability of the feature space by constructing negative sample comparisons within and between modalities. Overall, it achieves dual optimization of "alignment + discrimination" under unsupervised conditions, making the feature extraction network more discriminative and ultimately outputting robust multimodal feature representations.

[0101] In some embodiments, to further improve the stability of the trained feature extraction network, the network parameters of the feature extraction network are iteratively updated according to the cross-modal contrastive learning loss until a first preset training condition is met, including: Step B1: The electronic device determines the total target loss based on cross-modal contrastive learning loss, dynamic local binary cross-entropy, and gated conditional random field loss.

[0102] Building upon the existing cross-modal contrastive learning loss, this electronic device introduces dynamic local binary cross-entropy loss and gated conditional random field loss to jointly construct the total target loss. The cross-modal contrastive learning loss constrains the consistency of RGB and depth features in the semantic space, ensuring effective alignment of multimodal information. The dynamic local binary cross-entropy loss addresses the imbalance between foreground and background samples in salient target pseudo-labels by adaptively weighting samples of different categories within local regions. For example, when the salient target region in a video is small or samples are sparse in boundary regions, the weight of foreground pixels is increased to avoid bias towards background prediction. The gated conditional random field loss introduces spatial neighborhood constraints and combines them with gating mechanisms to suppress unreliable regions, making the salient target pseudo-labels spatially smoother and with clearer boundaries. For example, it strengthens consistency constraints between pixels at target edges and reduces unreasonable propagation in noisy regions. Through the synergistic effect of multiple losses, multimodal feature alignment is guaranteed, and salient target pseudo-labels are optimized from both the perspectives of category distribution and spatial structure.

[0103] Step B2: The electronic device iteratively updates the network parameters of the feature extraction network based on the total target loss until the total target loss is less than the first loss threshold.

[0104] The electronic device uses the total target loss as the optimization objective, and performs backpropagation and iterative updates on the network parameters of the feature extraction network. During training, as the dynamic local binary cross-entropy loss continuously balances the contributions of foreground and background samples, and the gated conditional random field loss continuously optimizes the consistency of the spatial structure, the feature representation generated by the model gradually stabilizes, thereby continuously improving the quality of the pseudo-labels for salient targets. When the total target loss drops below a preset first loss threshold, it is considered that the current network has fully learned discriminative and structurally consistent multimodal features, thus ending the training of the feature network in the first stage.

[0105] In this embodiment, by combining cross-modal contrastive learning with dynamic local binary cross-entropy and gated conditional random field constraints, effective alignment of multimodal features can be achieved on the one hand, and the quality of pseudo-labels of salient targets can be continuously optimized at the level of category distribution and spatial structure on the other hand. While balancing the number of positive and negative samples, the clarity of boundaries and consistency of regions can be improved, thereby providing more accurate and stable supervision signals for the subsequent training of video salient target detection models.

[0106] In some embodiments, to improve the quality of pseudo-labels for salient targets in videos, the electronic device is based on a pre-built multimodal online optimizer. This optimizer, driven by multimodal features, adaptively corrects the pseudo-labels for salient targets without relying on manual annotation, gradually evolving them from coarse responses to structurally clear and semantically consistent results, providing a foundation for subsequent fine-grained optimization. Specifically, each pseudo-label for a salient target in the video is iteratively updated, including: Step C1: For each pixel in the pseudo-label of the salient target in the video, the electronic device calculates the RGB feature distance, depth feature distance, and spatial location distance between the pixel and its neighboring pixels.

[0107] Step C2: The electronic device performs weighted fusion of the saliency responses of neighboring pixels based on RGB feature distance, depth feature distance, and spatial location distance to generate an updated saliency response.

[0108] Step C3: The electronic device iteratively updates the pseudo-labels of salient targets in the video based on the updated saliency response until the preset convergence condition is met.

[0109] For each pixel in a salient target pseudo-label, the electronic device can first calculate its RGB feature distance, depth feature distance, and spatial distance with neighboring pixels. The RGB feature distance characterizes color and texture similarity; for example, the surface of the same object usually has small RGB feature differences. The depth feature distance characterizes geometric consistency; for example, the same foreground target usually has continuous and gently changing depth. The spatial distance reflects the geometric proximity between pixels and is used to limit the range of information propagation. By introducing these three types of distances, the relationships between pixels can be characterized from three dimensions: appearance, structure, and space.

[0110] Then, the electronic device can perform weighted fusion of the saliency responses of neighboring pixels based on the three types of distances mentioned above to obtain an updated saliency response. Specifically, when neighboring pixels are similar to the current pixel in RGB and depth features and are spatially close, their saliency responses will receive higher weights and will be propagated preferentially; conversely, the influence of pixels that are significantly different in appearance or structure, or spatially distant, will be suppressed. Through this fusion mechanism, while maintaining consistency within the target area, information interference across targets or backgrounds can be effectively blocked, thereby achieving fine-tuning of the saliency response.

[0111] After obtaining the adjusted saliency response, the electronic device can iteratively update the salient target pseudo-labels based on the updated saliency response, and repeat the above process until a preset convergence condition is met (e.g., the update magnitude is lower than a threshold or the maximum number of iterations is reached). During multiple iterations, the salient region gradually becomes more complete and coherent, the boundaries become clearer, and the noise region is continuously weakened, thereby obtaining high-quality salient target pseudo-label results.

[0112] In this embodiment, the electronic device introduces RGB feature distance, depth feature distance, and spatial location distance during the optimization process of salient target pseudo-labels. On the one hand, it can achieve adaptive propagation of saliency information under multimodal and spatial constraints, enhancing the consistency within the target region. On the other hand, it can effectively suppress cross-regional interference and noise diffusion, thereby gradually improving the boundary accuracy and structural integrity of salient target pseudo-labels, and improving the overall quality of salient target pseudo-labels and the reliability of subsequent model training.

[0113] In some embodiments, the online optimizer is defined as:

[0114] The iterative optimization function is:

[0115] in, Represents the set of 8 neighboring pixels. and The weighting parameters are used to balance the importance of features. , and These are the RGB feature distance, depth feature distance, and positional distance, respectively. , and The corresponding standard deviation, Denotes the Hadamard product of matrices. For the first The video salient target pseudo-labels are updated in the next iteration. For the first The video salient target pseudo-labels are updated in the next iteration.

[0116] The aforementioned online optimizer achieves adaptive weighted propagation of saliency responses by introducing joint constraints of RGB feature distance, depth feature distance, and spatial location distance within the pixel neighborhood. Specifically, the exponential decay term based on feature distance grants higher weights to neighboring pixels similar in appearance and structure to the current pixel, thereby enhancing consistency within the same salient target in the video. The constraint based on location distance limits the spatial diffusion range of information, preventing mispropagation across regions. Simultaneously, parameters α1 and α2 balance neighborhood information with the current pixel's own response, ensuring that the update results possess both local consistency and maintain the stability of the original saliency distribution. Combined with a progressive iteration and Hadamard product update mechanism, the salient target pseudo-labels continuously strengthen high-confidence salient regions, suppress noise interference, and refine target boundaries during the iteration process, effectively improving the structural integrity, boundary clarity, and overall accuracy of the salient target pseudo-labels.

[0117] In some embodiments, in order to construct a reliable reference set and thus facilitate the effective fusion of long-term and short-term information, a reference set is constructed based on the current frame RGB-D data, its neighboring frame RGB-D data, and historical reference frames, including: Step D1: For each frame of historical RGB-D data within a preset time period, the electronic device calculates the quality score of the structural similarity between the pseudo-labels of significant targets corresponding to each frame of historical RGB-D data and the current frame of RGB-D data.

[0118] For each frame of historical RGB-D data within a preset time period, the electronic device calculates a quality score by comparing its structural similarity with the two salient target pseudo-labels corresponding to the current frame. The quality score quantifies the effectiveness of the historical frame. Specifically, this structural similarity considers factors such as region shape, boundary consistency, and saliency distribution. For example, when a target in a historical frame is unobstructed and has a clear outline, it has high structural similarity with the salient target pseudo-labels of the current frame, thus obtaining a high quality score; conversely, if the historical frame is blurry, obstructed, or contains false detections, its structural similarity is low.

[0119] The structural similarity between two salient target pseudo-labels can be calculated in a variety of ways.

[0120] For example, the Structural Similarity Index (SSIM) can be used for calculation. Specifically, the two salient target pseudo-labels are treated as grayscale images (or probability maps). The consistency of brightness, contrast, and structural information is calculated separately within a local window, and then weighted and fused to obtain the overall similarity score. For instance, if the salient target pseudo-label of the current frame and the salient target pseudo-label of a historical frame are relatively consistent in terms of the location distribution, boundary shape, and salience intensity of the target region, the SSIM value is close to 1; if there is a significant offset or structural distortion, the value decreases. This method can comprehensively reflect the consistency of regional distribution and local structure.

[0121] For example, the IoU can be calculated based on the degree of regional overlap, such as using the Intersection over Union (IoU). Specifically, after binarizing the pseudo-labels of salient targets, the ratio of the intersection to the union of the salient regions of the two frames can be calculated. If the salient target regions in the two frames of video highly overlap (e.g., the target positions and ranges are basically the same), the IoU value is high; if the target has undergone significant displacement or the detection is inconsistent, the IoU value is low. This method intuitively reflects the spatial consistency of the target regions.

[0122] For example, a boundary-based structural consistency metric can be used, such as extracting the edges of salient target pseudo-labels (e.g., through gradients or the Canny operator), and then calculating the distance or matching degree between the boundaries of two frames (e.g., Chamfer distance or F-measure). For instance, the boundary matching degree is high when the target outlines in two frames are clear and have consistent shapes; if there is blurring or deformation, the matching degree decreases. This method is quite sensitive to changes in target shape.

[0123] In addition, feature-level methods can be combined, such as mapping salient target pseudo-labels to the feature space (e.g., extracting high-level semantic features through a convolutional encoder), and then calculating the cosine similarity between feature vectors, thereby measuring structural consistency at the semantic level.

[0124] In practical applications, one of the above methods can be selected, or multiple similarity indicators can be weighted and fused to obtain a more robust structural similarity quality score, thereby providing a reliable basis for the selection of historical reference frames.

[0125] Step D2: The electronic device selects at least one historical RGB-D data frame with the highest quality score from each frame of historical RGB-D data as the historical reference frame RGB-D data for the current frame RGB-D data.

[0126] Based on the aforementioned quality scores, at least one frame with the highest score is selected from historical RGB-D data as the historical reference frame for the current frame. This selection process essentially chooses the "most representative" and "most reliable" moment in a long-term sequence as the source of long-term memory information. For example, if a target is briefly occluded, historical frames from before or without occlusion can be selected as references, thus providing more complete information about the target's appearance and structure for the current frame. This selection mechanism effectively avoids low-quality historical frames interfering with subsequent modeling.

[0127] Step D3: The electronic device constructs a reference set based on the historical reference frame RGB-D data and the specified number of adjacent frames corresponding to the current frame RGB-D data.

[0128] The selected historical reference frames, together with a specified number of neighboring frames of the current frame, constitute a reference set. The neighboring frames provide short-term, continuous change information, such as the target's local motion trajectory and deformation; the historical reference frames provide long-term stable information, such as the target's complete structure under ideal conditions. By constructing this reference set, a unified input is provided for subsequent temporal feature fusion, enabling the model to simultaneously access both short- and long-term information sources.

[0129] In this embodiment, by performing quality assessment and screening of historical frames based on structural similarity, high-quality reference information can be extracted from long-term time series and used together with short-term adjacent frames to participate in feature modeling. This enables an effective combination of long-term stability and short-term dynamism in subsequent time series fusion processes, improving the model's adaptability to occlusion, deformation and complex dynamic scenes, and enhancing the temporal consistency and overall robustness of video salient target detection.

[0130] In some embodiments, during the training of the salient object detection model, multiple sets of binary cross-entropy losses can be introduced to jointly supervise the feature outputs at different levels and scales, thereby improving the multi-scale feature extraction and fusion effect. Specifically, auxiliary output branches are set in each key layer of the encoder and decoder of the model (e.g., shallow detail features, mid-level semantic features, and high-level global features) to generate saliency prediction maps at different resolutions, and binary cross-entropy loss is calculated for the prediction results of each layer. The shallow branches focus on constraining boundary details and local structure to make the target outline clearer; the mid-level branches strengthen regional consistency and semantic transition; and the high-level branches focus on the overall saliency distribution and global semantic expression.

[0131] Building upon this foundation, weighted coefficients can be introduced into the losses of each layer for weighted fusion to balance the contributions of different layers to the final result. For example, higher-level semantics can be given greater weight to ensure overall detection accuracy, while shallow constraints are retained to refine the boundaries. Through the collaborative optimization of multiple loss sets, the model can simultaneously focus on global semantics and local details during training, thereby improving the fusion capability of multi-scale features and avoiding the information loss problem caused by single-scale supervision.

[0132] Multiple sets of binary cross-entropy loss can explicitly guide the feature learning process at different levels, effectively constraining features at all scales, enhancing the hierarchy and complementarity of feature expression, thereby improving the completeness, boundary accuracy and overall robustness of video salient target detection results.

[0133] In some embodiments, a salient object detection method is illustrated by way of an example.

[0134] First, in the data preparation stage, the DViSal dataset was selected as the source of training and testing data. Each frame in the video sequence was preprocessed, the input image was uniformly adjusted to 320×320 resolution, and the corresponding RGB image and depth image were extracted as model input, thereby constructing a data foundation with multi-scene and multi-modal features.

[0135] In the first stage of training, a pre-trained MOCO model is used as the feature extraction network to extract features from RGB and depth images respectively, obtaining RGB and depth feature maps. Corresponding RGB and depth activation maps are then generated using activation heads constructed from single-layer convolutions. Based on this, intra-sample contrast saliency learning is implemented. This involves calculating the consistency between foreground and background representations and setting the hyperparameter α to 0.5 to strengthen the consistency constraints between salient and non-salient regions across different modalities in a weighted manner. Simultaneously, inter-sample contrast saliency learning is combined with constructing intra-modal and inter-modal negative sample losses to increase the feature distance between positive and negative samples, thereby improving feature discrimination ability. Subsequently, a multi-modal online optimization mechanism is introduced to iteratively update the pseudo-labels of salient targets. The formula for the online optimizer is as follows:

[0136] Where σ rgb =0.1、σ dep =0.1、σ p =0.05, α1=0.6, α2=0.4, and the number of iterations is 3, in order to integrate local neighborhood information and refine the structure of significant regions.

[0137] Furthermore, dynamic stability training is performed based on the following objective function:

[0138]

[0139] The two super parameters α and β are switched at different training stages, and the threshold parameters θ_high and θ_low are adjusted linearly with the number of training rounds. The batch size is set to 8, and the total number of training rounds is 20, thereby improving the stability and convergence of the training process.

[0140] In the second stage of training, firstly, the quality score between the salient target pseudo-label of the current frame and the salient target pseudo-label of each historical frame is calculated based on structural similarity scoring, and historical reference frames with optimal parameters for the current frame are selected. Based on this, the recent historical window size K is set to 3, and the historical reference frames and the three adjacent frames of the current frame are used to construct a salient frame reference set to fuse long-term and short-term temporal information, obtaining temporal fusion features. Subsequently, during model training, multiple sets of binary cross-entropy loss are used to jointly constrain features at different levels and scales to improve feature fusion performance. Simultaneously, the Adam optimizer is used for parameter updates, with a learning rate of 0.0001, a batch size of 2, and a total of 50 training epochs, thus obtaining the final video salient target detection model.

[0141] The training method in this embodiment can train a salient object detection model under unsupervised conditions, demonstrating good practicality and promotional value. Firstly, by constructing a training framework combining cross-modal contrastive learning and temporal feature fusion, salient information is automatically mined without any manually labeled data, significantly reducing data labeling costs and improving the feasibility of the training method in real-world scenarios. Adaptive cross-modal contrastive learning, through the synergistic effect of intra- and inter-sample contrastive learning and multimodal online optimization, fully explores the consistency and complementary features between RGB and deep modalities, effectively alleviating the problem of insufficient modality fusion in traditional methods and improving feature representation capabilities.

[0142] Meanwhile, by introducing an unsupervised dual-memory fusion mechanism, the long-term stable information represented by historical reference frames and the short-term dynamic information represented by adjacent frames are jointly modeled to obtain temporal fusion features. Training the salient target detection model with these features allows the model to fully utilize the long-range and short-range temporal dependencies in the video sequence, thereby significantly improving the temporal consistency and stability of the detection results. Extensive experimental results show that the model trained in this way not only outperforms existing unsupervised methods but also achieves high-precision detection in various complex indoor and outdoor scenes, with performance approaching that of mainstream fully supervised methods. Furthermore, due to the good versatility of the proposed cross-modal saliency modeling method, it can be further transferred to other multimodal video salient target detection tasks such as RGB-T, demonstrating strong scalability and broad application prospects.

[0143] In some embodiments, the salient object detection model obtained after training differs from the model during the training process. The following section uses a specific application scenario to illustrate the operating mechanism of the model after training and compares it with the training process.

[0144] For example, when a robot performs indoor navigation tasks, it needs to detect salient objects in real-time input video frames to assist path planning. During model training, the input consists of pairs of RGB and depth images. The model employs a dual-branch structure to extract RGB and depth features separately, and constructs contrastive learning constraints through an adaptive cross-modal saliency mechanism (ACMS). In this process, ACMS exists in the computational graph as a loss function. Through intra- and inter-sample contrast constraints, it continuously forces RGB features to align with depth features, enabling the RGB branch to gradually learn the spatial structure and geometric information contained in the depth modality. Essentially, this stage is a "constrained feature learning process." ACMS participates in the backpropagation during feature extraction network training as a training aid mechanism, but it is not part of the feature extraction network itself.

[0145] When the model is trained and deployed in a real-world application, its operating mechanism changes significantly. At this point, only the trained feature extraction network and subsequent detection modules are retained, while the ACMS-related loss calculation logic and depth branches are removed. Taking a navigation scenario as an example, the system only needs to input an RGB image during the inference phase, and the model can directly output the detection results of salient objects in the video, without needing to input a depth image or perform cross-modal contrast calculations. This means that the complex computational path of "dual-modal input + contrast constraints" in the training phase is simplified into a streamlined process of "modal forward inference" during the inference phase.

[0146] From a computational perspective, this difference is not only reflected in the parameters but also in the change in the computational graph structure: during training, the ACMS loss acts as a key logical node connecting the RGB and depth branches and participates in parameter updates; however, during inference, this node is completely removed, and the model retains only the optimized feature extraction capability. Therefore, the inference process can be viewed as a functionally simplified version of the training process, inheriting the high-quality multimodal representation capabilities learned during training while eliminating all auxiliary components used for supervision and constraints.

[0147] Through the above mechanism, the model can achieve video salient object detection with near-multimodal training effect by relying only on RGB input in practical applications. On the one hand, it reduces the dependence on sensors and computational overhead in the inference stage, and on the other hand, it still maintains high detection accuracy and robustness.

[0148] In some embodiments, the robot may deploy a trained video salient object detection model to perform salient object detection based on the model: Step D1: The robot acquires the target RGB-D video sequence collected by the robot in performing the current task.

[0149] During the execution of its current task, the robot can acquire target RGB-D video sequences in real time through its onboard vision sensors, or receive target RGB-D video sequences uploaded by the user as input. For example, in indoor navigation tasks, the robot uses cameras and depth sensors to continuously collect RGB images and corresponding depth information of the environment; in grasping tasks, it acquires the appearance features and spatial structure information of target objects within the operating area; in automated inspection or video surveillance tasks, it continuously acquires multimodal video of dynamic scenes, thereby providing multimodal perception data with temporal continuity and spatial structure information for subsequent salient target detection in videos, thus supporting accurate analysis and decision-making in complex environments.

[0150] Step D2: The robot inputs the target RGB-D video sequence into the video salient target detection model to obtain the salient target detection results.

[0151] The robot inputs the acquired RGB-D video sequence into a pre-trained video salient object detection model to obtain the corresponding salient object detection results. Specifically, the robot can process each frame of the input video based on the model and output the region distribution or probability map of salient objects in the video. For example, in navigation scenarios, the model can highlight salient objects such as pedestrians and obstacles; in grasping scenarios, it can highlight the object to be grasped; and in video editing or monitoring scenarios, it can highlight moving subjects or key objects, thereby achieving automatic extraction of semantic video salient objects from raw visual data.

[0152] Step D3: Based on the salient target detection results and the robot's current task status, the robot controls its operation.

[0153] The robot controls its behavior based on the detection results of salient targets and the current task status. For example, in navigation tasks, the robot can adjust its path to avoid obstacles based on the position of salient targets (such as obstacles or pedestrians) in the video; in grasping tasks, it can locate the target object and plan the grasping action based on the salient target area in the video; in inspection tasks, it can prioritize the detection or alarm of saliently abnormal areas. By combining the detection results with the task decision logic, a closed-loop control from perception to decision and then to execution is achieved.

[0154] In this embodiment, by deploying a video salient target detection model in the robot system, the robot can automatically identify key targets in complex and dynamic environments and make adaptive decisions and controls based on the detection results, thereby improving environmental perception and task execution efficiency, and achieving more intelligent and robust autonomous operation.

[0155] In some embodiments, see Figure 2 , Figure 2 A schematic diagram of a video salient object detection model structure is shown. The model can be divided into two core stages: "multimodal representation learning and salient object pseudo-label generation (upper part)" and "temporal modeling and detection model training (lower part)," reflecting the complete technical path from unsupervised representation learning to temporal enhanced detection.

[0156] In the first part, the input is an RGB-D video sequence, which is processed by a multimodal encoder to extract RGB features and depth features respectively. Then, a decoupler decomposes the two modal features into foreground and background features (RGB foreground / background, depth foreground / background), and feature alignment is performed under contrastive learning constraints. A mechanism of "positive samples being brought closer and negative samples being separated" is used to strengthen cross-modal consistency and discriminability. Based on this, the multimodal features are fused to generate a cross-modal class activation map (CAM), resulting in an initial salient response map. Further refinement is performed to optimize the salient response, generating salient target pseudo-labels, which are then processed by the encoder to obtain optimized salient target pseudo-label representations. This stage essentially completes unsupervised multimodal feature learning and high-quality salient target pseudo-label construction.

[0157] In the second half, a temporal modeling mechanism is introduced based on the fusion results of RGB and depth features. First, for the current frame, several adjacent frames (T-1, T-2, T-3, etc.) and high-quality historical reference frames are selected from the historical sequence to construct a reference set. Simultaneously, the features of each frame are stored in a memory feature library for cross-temporal information retrieval and reuse. The decoder receives the features of the current frame and combines them with the historical reference features extracted from the memory feature library, integrating temporal information through feature interaction and fusion. Finally, the "decoding and fusion" module outputs the salient target detection results.

[0158] Overall, the model improves the quality of multimodal feature representation through cross-modal contrastive learning and optimization of salient target pseudo-labels in the previous stage. In the next stage, by introducing dual temporal information modeling of "historical reference frame + adjacent frame" and combining it with the memory feature library mechanism, it achieves effective fusion of long and short-term temporal features, thereby enhancing the model's adaptability to dynamic changes, occlusion and complex backgrounds, and ensuring the temporal consistency and robustness of video salient target detection results.

[0159] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0160] Corresponding to the training method of the video salient object detection model in the above embodiment, Figure 3 The diagram shows a structural block diagram of the training device 3 for the video salient target detection model provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0161] Reference Figure 3 The training device 3 for the video salient object detection model includes: The acquisition module 31 is used to acquire RGB-D video sequences collected when the robot performs a preset task, so as to obtain RGB-D data for each frame; each frame of RGB-D data includes an RGB image and a corresponding depth image; The first-stage training module 32 is used to input each frame of RGB image and the corresponding frame of depth image into the feature extraction network to obtain each RGB feature and each depth feature, and to construct a cross-modal contrastive learning loss based on each RGB feature and each depth feature. The network parameters of the feature extraction network are iteratively updated according to the cross-modal contrastive learning loss until the first preset training condition is met. The optimization module 33 is used to generate pseudo-labels for each salient target corresponding to each frame of RGB-D data based on the updated network parameters after the first preset training conditions are met, and to iteratively update each salient target pseudo-label based on each RGB feature, each depth feature and the corresponding pixel spatial neighborhood relationship within a preset neighborhood range. The second-stage training module 34 is used to train the video salient target detection model with the iteratively updated pseudo-labels of each salient target as the supervision signal. It constructs a reference set based on the RGB-D data of the current frame and its neighboring frames and historical reference frames, and performs temporal feature fusion through the reference set to train the video salient target detection model until the second preset training condition is met to obtain the trained video salient target detection model.

[0162] Optionally, the first-stage training module 32 includes a loss calculation unit, which is used for: Activation operations are performed on each RGB feature map and each depth feature map to obtain each RGB activation map and each depth activation map; The intra-sample contrast significance loss and inter-sample contrast significance loss are calculated based on each RGB feature map, each depth feature map, each RGB activation map, and each depth feature map. The modal contrast learning loss is determined based on the intra-sample contrast significance loss and the inter-sample contrast significance loss.

[0163] Optionally, the loss calculation unit is specifically used for: For each frame of RGB-D data, RGB foreground and background representations are determined based on the corresponding RGB feature map and RGB activation map; depth foreground and depth background representations are determined based on the corresponding depth feature map and depth activation map; and foreground activation is calculated based on the RGB foreground and depth foreground representations. Figure 1 Consistency; Calculating background activation based on RGB background representation and depth background representation Figure 1 To the point of being compatible; Based on each foreground activation Figure 1 Consistency determines the positive sample loss of the prospect; Activation based on various backgrounds Figure 1 Consistency determination of background positive sample loss; The in-sample contrast significance loss is determined based on the foreground positive sample loss and the background positive sample loss; The intermodal negative sample loss is determined based on the RGB foreground representation and the background representation at each depth. Intramodal negative sample loss is determined based on each RGB foreground representation and each RGB background representation; The significance loss between samples is determined based on the negative sample loss between modalities and the negative sample loss within modalities.

[0164] Optionally, the loss calculation unit is specifically used for: Activate based on each foreground Figure 1 Homogeneity is activated in various foregrounds Figure 1 The ranking in consistency determines the corresponding dynamic weights of the foreground; based on the activation of each foreground... Figure 1 Consistency and corresponding dynamic weights are used to calculate the loss of positive prospect samples; Activation based on various backgrounds Figure 1 Consistency determination of background positive sample loss includes: Activate based on each background Figure 1 Activation of homogeneity in various backgrounds Figure 1 Ranking in consistency determines background dynamic weights; activation based on each background Figure 1 Consistency and corresponding background dynamic weights are used to calculate the background positive sample loss.

[0165] Optionally, the first-stage training module 32 includes a first training unit, which is used for: The total target loss is determined based on cross-modal contrastive learning loss, dynamic local binary cross-entropy, and gated conditional random field loss. The network parameters of the feature extraction network are iteratively updated based on the target total loss until the target total loss is less than the first loss threshold.

[0166] Optionally, optimization module 33 is specifically used for: Based on a pre-built multimodal online optimizer, the pseudo-labels for each salient target are iteratively updated: For each pixel in the salient target pseudo-label, calculate the RGB feature distance, depth feature distance, and spatial location distance between the pixel and its neighboring pixels; The saliency responses of neighboring pixels are weighted and fused based on RGB feature distance, depth feature distance, and spatial location distance to generate an updated saliency response; The pseudo-labels of salient targets are iteratively updated based on the updated salient responses until the preset convergence condition is met.

[0167] Optionally, the online optimizer is defined as:

[0168] The iterative optimization function is:

[0169] in, Represents the set of 8 neighboring pixels. and The weighting parameters are used to balance the importance of features. , and These are the RGB feature distance, depth feature distance, and positional distance, respectively. , and The corresponding standard deviation, Denotes the Hadamard product of matrices. For the first The salient target pseudo-labels updated in the next iteration. For the first The salient target pseudo-labels are updated in the next iteration.

[0170] Optionally, the second-stage training module 34 is specifically used for: For each frame of historical RGB-D data within a preset time period, calculate the quality score of the structural similarity between the pseudo-labels of significant targets corresponding to each frame of historical RGB-D data and the current frame of RGB-D data. Select at least one historical RGB-D data frame with the highest quality score from each frame of historical RGB-D data as the historical reference frame RGB-D data for the current frame RGB-D data; A reference set is formed by combining the RGB-D data of historical reference frames with a specified number of adjacent frames corresponding to the RGB-D data of the current frame.

[0171] Figure 4 This is a schematic diagram of the physical layer structure of an electronic device provided in an embodiment of this application. Figure 4As shown, the electronic device 4 of this embodiment includes: at least one processor 41 ( Figure 4 Only one processor is shown in the diagram), memory 42, and computer program 43 stored in memory 42 and executable on at least one processor 41. When processor 41 executes computer program 43, it implements the steps in the above-described embodiments of the training method for any video salient object detection model, for example... Figure 1 Steps 110-150 are shown.

[0172] The processor 41 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0173] In some embodiments, memory 42 may be an internal storage unit of electronic device 4, such as a hard disk or memory of electronic device 4. In other embodiments, memory 42 may also be an external storage device of electronic device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 4.

[0174] Furthermore, the memory 42 may include both internal storage units and external storage devices of the electronic device 4. The memory 42 is used to store operating devices, application programs, bootloaders, data, and other programs, such as program code for computer programs. The memory 42 can also be used to temporarily store data that has been output or will be output.

[0175] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0176] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0177] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0178] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0179] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0180] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0181] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0182] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A training method for a video salient object detection model, characterized in that, The training method includes: Acquire RGB-D video sequences captured by the robot while it is performing a preset task to obtain RGB-D data for each frame; each frame of RGB-D data includes an RGB image and a corresponding depth image; Each frame of the RGB image and the corresponding frame of the depth image are input into the feature extraction network to obtain each RGB feature and each depth feature. A cross-modal contrastive learning loss is constructed based on each RGB feature and each depth feature. The network parameters of the feature extraction network are iteratively updated according to the cross-modal contrastive learning loss until the first preset training condition is met. After meeting the first preset training conditions, pseudo-labels for each video salient target are generated based on the updated network parameters and the RGB-D data of each frame are updated iteratively within a preset neighborhood based on each RGB feature, each depth feature and the corresponding pixel spatial neighborhood relationship. The video salient target detection model is trained using the iteratively updated pseudo-labels of each video salient target as the supervision signal. A reference set is constructed based on the RGB-D data of the current frame, the RGB-D data of its neighboring frames, and historical reference frames. Temporal features are fused through the reference set to train the video salient target detection model until the second preset training condition is met, thus obtaining the trained video salient target detection model.

2. The training method as described in claim 1, characterized in that, The construction of cross-modal contrastive learning loss based on each of the RGB features and each of the deep features includes: Activation operations are performed on each of the RGB feature maps and each of the depth feature maps to obtain each RGB activation map and each depth activation map; The intra-sample contrast significance loss and inter-sample contrast significance loss are calculated based on each of the RGB feature maps, each of the depth feature maps, each of the RGB activation maps, and each of the depth feature maps. The modality contrast learning loss is determined based on the intra-sample contrast significance loss and the inter-sample contrast significance loss.

3. The training method as described in claim 2, characterized in that, The calculation of intra-sample contrast significance loss and inter-sample contrast significance loss based on each of the RGB feature maps, each of the depth feature maps, each of the RGB activation maps, and each of the depth feature maps includes: For each frame of RGB-D data, RGB foreground and RGB background representations are determined based on the corresponding RGB feature map and the corresponding RGB activation map; depth foreground and depth background representations are determined based on the corresponding depth feature map and the corresponding depth activation map; foreground activation map consistency is calculated based on the RGB foreground representation and the depth foreground representation; background activation map consistency is calculated based on the RGB background representation and the depth background representation. The foreground positive sample loss is determined based on the consistency of the foreground activation maps described above. The background positive sample loss is determined based on the consistency of the background activation maps described above; The intra-sample contrast significance loss is determined based on the foreground positive sample loss and the background positive sample loss; The intermodal negative sample loss is determined based on the RGB foreground representation and the background representation at each depth. Intramodal negative sample loss is determined based on each RGB foreground representation and each RGB background representation; The inter-modal negative sample loss and the intra-modal negative sample loss are used to determine the significance loss of the inter-sample comparison.

4. The training method as described in claim 1, characterized in that, The determination of the positive foreground sample loss based on the consistency of each of the foreground activation maps includes: The corresponding dynamic weight of the foreground is determined based on the ranking of the consistency of each foreground activation graph in the consistency of each foreground activation graph. The foreground positive sample loss is calculated based on the consistency of each foreground activation map and the corresponding dynamic weights. The determination of the background positive sample loss based on the consistency of each of the background activation maps includes: The background dynamic weight is determined based on the ranking of the consistency of each background activation graph in the consistency of each background activation graph. The background positive sample loss is calculated based on the consistency of each background activation map and the corresponding background dynamic weights.

5. The training method as described in any one of claims 1-4, characterized in that, The step of iteratively updating the network parameters of the feature extraction network based on the cross-modal contrastive learning loss until the first preset training condition is met includes: The total target loss is determined based on the cross-modal contrastive learning loss, dynamic local binary cross-entropy, and gated conditional random field loss. The network parameters of the feature extraction network are iteratively updated based on the target total loss until the target total loss is less than a first loss threshold.

6. The training method according to any one of claims 1-4, characterized in that, The iterative update of the pseudo-labels for each salient target in the video within a preset neighborhood based on each of the RGB features, each of the depth features, and the corresponding pixel spatial neighborhood relationships includes: Based on a pre-built multimodal online optimizer, the pseudo-labels for each video's salient targets are iteratively updated: For each pixel in the pseudo-label of the salient target in the video, calculate the RGB feature distance, depth feature distance, and spatial location distance between the pixel and its neighboring pixels; The saliency responses of neighboring pixels are weighted and fused based on the RGB feature distance, depth feature distance, and spatial location distance to generate an updated saliency response; The pseudo-labels of the salient targets in the video are iteratively updated based on the updated salient response until a preset convergence condition is met.

7. The training method as described in claim 6, characterized in that, The online optimizer is defined as: The iterative optimization function is: in, Represents the set of 8 neighboring pixels. and The weighting parameters are used to balance the importance of features. , and These are the RGB feature distance, depth feature distance, and positional distance, respectively. , and The corresponding standard deviation, Denotes the Hadamard product of matrices. For the first The video salient target pseudo-labels are updated in the next iteration. For the first The video salient target pseudo-labels are updated in the next iteration.

8. The training method as described in any one of claims 1-4, characterized in that, The construction of a reference set based on the current frame's RGB-D data, its neighboring frames' RGB-D data, and historical reference frames includes: For each frame of historical RGB-D data within a preset time period, calculate the quality score of the structural similarity between the historical RGB-D data of each frame and the salient target pseudo-labels corresponding to the RGB-D data of the current frame; Select at least one historical RGB-D data frame with the highest quality score from each frame of historical RGB-D data as the historical reference frame RGB-D data of the current frame RGB-D data; A reference set is formed by the historical reference frame RGB-D data and the specified number of adjacent frames corresponding to the current frame RGB-D data.

9. A method for detecting salient targets, characterized in that, The salient target detection method, applied to robots, includes: Acquire the target RGB-D video sequence acquired by the robot during the current task; The target RGB-D video sequence is input into the video salient target detection model to obtain the salient target detection results; Based on the salient target detection results and the robot's current task status, the robot is controlled to operate. The video salient object detection model is obtained after training using the model training method described in any one of claims 1 to 8.

10. A training device for a video salient object detection model, characterized in that, include: The acquisition module is used to acquire RGB-D video sequences collected when the robot performs a preset task, so as to obtain RGB-D data for each frame; Each frame of RGB-D data includes an RGB image and a corresponding depth image; The first-stage training module is used to input the RGB images of each frame and the corresponding depth images of each frame into the feature extraction network to obtain each RGB feature and each depth feature, and to construct a cross-modal contrastive learning loss based on each RGB feature and each depth feature, and to iteratively update the network parameters of the feature extraction network according to the cross-modal contrastive learning loss until the first preset training condition is met. The optimization module is used to generate pseudo-labels for each video salient target corresponding to each frame of RGB-D data based on the updated network parameters after the first preset training conditions are met, and to iteratively update each video salient target pseudo-label based on each RGB feature, each depth feature and the corresponding pixel spatial neighborhood relationship within a preset neighborhood range; The two-stage training module is used to train the video salient target detection model with the iteratively updated pseudo-labels of each video salient target as the supervision signal. The module constructs a reference set based on the RGB-D data of the current frame, the RGB-D data of its neighboring frames, and historical reference frames. Temporal feature fusion is performed through the reference set to train the video salient target detection model until the second preset training condition is met to obtain the trained video salient target detection model.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training method as described in claims 1-8.

12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method as described in any one of claims 1 to 8.

Images