A moving target segmentation method and system based on satellite video

Abstract

The present invention provides a method and system for moving target segmentation based on satellite video. The method first determines a reference frame image based on a query frame image to be segmented, and determines a corresponding true mask image based on the reference frame image. Then, shape prior extraction is performed based on the true mask image to determine an edge mask image. Finally, a moving target segmentation model is used to perform feature extraction, spatiotemporal semantic relationship modeling, and semantic affinity constraint processing on the query frame image, reference frame image, true mask image, and edge mask image, and outputs a target mask image for representing the moving target segmentation result. In this way, the method can achieve efficient interactive learning of semantic information within and between satellite video frames, improve the robustness and accuracy of semantic learning of the moving target segmentation model, and significantly enhance the edge feature expression capability, thereby effectively improving the efficiency and accuracy of moving target segmentation in satellite videos.

Description

Technical Field

[0001] The present invention relates to the technical field of satellite video processing and analysis, and in particular to a moving target segmentation method and system based on satellite video. Background Art

[0002] Video satellites are a new type of Earth observation satellite. Compared to traditional Earth observation satellites, video satellites can continuously observe a specific area, thereby acquiring remote sensing imagery with high spatial and spectral resolution, as well as high temporal resolution. Intelligent processing and analysis of satellite video can automatically extract and analyze information from scenes of interest, providing important technical support for practical applications such as disaster monitoring, ocean monitoring, and ecosystem disturbance monitoring. Satellite video object segmentation, in particular, enables fine-grained segmentation of objects of interest in time-continuous scenes. This not only locates the target's position but also provides additional, detailed descriptive information, supporting more sophisticated analysis and applications.

[0003] For satellite videos, the technologies related to moving target segmentation are mainly divided into two categories: unsupervised and semi-supervised. Specifically, methods based on optical flow combined with unsupervised learning can capture the motion characteristics of the target in the temporal dimension through optical flow information, but such methods are highly dependent on the accuracy of optical flow estimation. When complex scenes appear in satellite videos (such as occlusion, lighting changes, or rapid movement of targets), optical flow estimation is prone to failure, resulting in a decrease in segmentation performance. Supervised learning methods based on static image segmentation, such as OSVOS (One-Shot Video Object Segmentation), rely on the annotation of the first frame to fine-tune the model, and only use static features to process subsequent frames frame by frame, ignoring the temporal information of the video, making it difficult to effectively deal with changes in target appearance or background dynamic interference. In addition, these methods generally have the problems of high computational overhead and lack of real-time performance, which limits their application on resource-constrained satellite payloads.

[0004] It can be seen that the relevant methods for moving target segmentation based on satellite videos have problems such as complex calculation process, insufficient refinement of target segmentation edges, and insufficient mining and utilization of temporal information, which makes it impossible to achieve moving target segmentation efficiently and accurately. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that it is impossible to efficiently and accurately segment moving targets based on satellite video.

[0006] To solve the above technical problems, the present invention provides a moving target segmentation method and system based on satellite video, which specifically adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for moving target segmentation based on satellite video, comprising: first, determining a reference frame image based on a query frame image to be segmented in a satellite video, and determining a corresponding true mask image based on the reference frame image. The reference frame image is a frame image in the satellite video that has been segmented, and the true mask image is used to characterize the position and shape of the moving target in the reference frame image. Then, shape prior extraction is performed based on the true mask image to determine an edge mask image, which is used to characterize the edge shape of the moving target. Secondly, feature extraction, spatiotemporal semantic relationship modeling, and semantic affinity constraint processing are performed on the query frame image, the reference frame image, the true mask image, and the edge mask image using a moving target segmentation model, and a target mask image is output, which is used to characterize the moving target segmentation result.

[0008] This method significantly enhances the expressiveness of edge features through shape prior fusion. Furthermore, to address the interference of stationary objects on the semantic learning of the moving target segmentation model, this method employs foreground-background semantic affinity constraints to improve the robustness and accuracy of semantic learning of the moving target segmentation model. This method, through feature extraction, spatiotemporal semantic relationship modeling, and semantic affinity constraint processing within the moving target segmentation model, can achieve precise segmentation of moving targets in satellite videos and effectively adapt to complex backgrounds and lighting changes, thereby effectively improving the efficiency and accuracy of moving target segmentation in satellite videos.

[0009] In combination with the first aspect, in an optional implementation method, the above-mentioned motion target segmentation model includes: a feature extraction module, a spatiotemporal semantic relationship modeling module and a semantic affinity constraint segmentation decoding head module. Among them, the feature extraction module can be used to respectively extract features for characterizing the spatial information and edge characteristics of the object in the query frame image, the reference frame image, the real mask image and the edge mask image, and obtain the corresponding first extracted features, second extracted features, third extracted features and fourth extracted features, and splice the first extracted features, the second extracted features, the third extracted features and the fourth extracted features to determine the target splicing features. The spatiotemporal semantic relationship modeling module can be used to perform position encoding and addition, spatiotemporal semantic relationship modeling encoding and target prediction decoding according to the target splicing features to determine the encoding features and decoding features. The semantic affinity constraint segmentation decoding head module can be used to perform mask prediction based on the encoding features and decoding features to determine the target mask image.

[0010] In this implementation, the moving target segmentation model can accurately extract features used to characterize the spatial information and edge characteristics of the object from the query frame image, reference frame image, true mask image, and edge mask image through the feature extraction module. By constructing spatiotemporal semantic relationships through the spatiotemporal semantic relationship modeling module, the moving target segmentation model can effectively process the spatiotemporal information of the moving target, solve the diversity and uncertainty problems in the moving target segmentation, and thus improve the accuracy of the moving target segmentation. The semantic affinity constraint segmentation decoding head module can realize robust representation learning of semantic consistency, which can effectively improve the robustness and accuracy of the moving target segmentation model for semantic learning of moving targets.

[0011] In combination with the first aspect, in an optional implementation, the above-mentioned spatiotemporal semantic relationship modeling module includes: a position encoding module, a Transformer encoder and a Transformer decoder. Among them, the position encoding module can be used to add position encoding to the target splicing feature to obtain the position encoding splicing feature. The Transformer encoder can be used to perform spatiotemporal semantic relationship modeling encoding through a multi-head self-attention module and a fully connected feedforward neural network based on the position encoding splicing feature, and determine and output the encoding feature. The Transformer decoder can be used to perform target prediction decoding through a multi-head self-attention module, a multi-head cross-attention module and a fully connected feedforward neural network based on the encoding feature, and determine and output the decoding feature.

[0012] In this implementation, position encoding is introduced through the position encoding module to provide position information, so that the Transformer encoder can effectively capture the spatiotemporal dependency between pixels in the input frame through spatiotemporal semantic relationship modeling encoding. The moving target segmentation model can model the dynamic changes between objects in multiple time steps, thereby improving the accuracy of target detection and tracking. The Transformer encoder can also learn and establish the spatial-temporal correspondence between the query frame image and the reference frame image through spatiotemporal semantic relationship modeling encoding. The Transformer decoder can predict the spatial position information of the moving target in each frame image through target prediction decoding, thereby enhancing the spatiotemporal reasoning ability of the moving target segmentation model. In this way, not only can the transformation of the moving target between different time steps be accurately captured, but also by learning the robust representation of the target object, the moving target segmentation model can improve the accuracy of moving target segmentation when facing the confusion problem of similar objects.

[0013] In conjunction with the first aspect, in one optional implementation, the semantic affinity constrained segmentation decoding head module includes: a target affinity module and a target mask prediction module. The target affinity module can be configured to determine and output a self-attention feature map based on the decoded features and the encoded features. The target mask prediction module can be configured to determine a target mask image based on the self-attention feature map and the encoded features through upsampling and convolution.

[0014] In this implementation, the target affinity module and the target mask prediction module can explicitly constrain the learning process of the moving target segmentation model, and enable the moving target segmentation model to more accurately distinguish moving targets from stationary targets and optimize the semantic consistency of the background and foreground.

[0015] In conjunction with the first aspect, in an optional implementation, the semantic affinity constrained segmentation decoding head module further includes a semantic affinity prediction module. The semantic affinity prediction module can be configured to generate a semantic affinity map based on the self-attention feature map through convolution, batch normalization, and sigmoid activation. The semantic affinity map is a heat map representing the semantic relevance between foreground and background, and is used to train a moving object segmentation model.

[0016] In this implementation, during the training process of the motion target segmentation model, a semantic affinity graph can be generated by the semantic affinity prediction module to provide the motion target segmentation model with strong supervision of the semantic relationship between pixels, thereby optimizing the accuracy and robustness of the motion target segmentation model in segmenting motion targets.

[0017] In combination with the first aspect, in an optional implementation, the loss function of the above-mentioned moving target segmentation model is expressed as:

[0018] ;

[0019] ;

[0020] ;

[0021] ;

[0022] in, represents the loss function of the moving target segmentation model, represents the classification loss of pixels in the frame image, represents the target mask prediction loss, Indicates affinity loss, represents the set of pixels in the target mask image, Indicates the target of movement The predicted mask image, Indicates the target of movement The true mask image of Indicates the number of moving targets, represents the size of the semantic affinity graph, Represents the feature points of the semantic affinity graph, represents the predicted probability of the semantic affinity graph, Represents the true value of the target probability distribution. Indicates the sports goals, Indicates the The target mask image corresponding to the moving target, Indicates the target mask image at pixel point The value at Indicates the true value mask image at pixel point The value at .

[0023] In conjunction with the first aspect, in an optional implementation, performing shape prior extraction based on the true mask image to determine the edge mask image includes: first, preprocessing the true mask image to obtain a preprocessed true mask image; then, performing mask binarization on the preprocessed true mask image to obtain a binarized mask image; and finally, performing edge detection on the binarized mask image to obtain an edge mask image.

[0024] In conjunction with the first aspect, in an optional implementation, performing edge detection on the binarized mask image to obtain the edge mask image includes: first, determining horizontal gradients and vertical gradients in the image based on the binarized mask image; then, determining gradient amplitudes based on the horizontal gradients and vertical gradients; then, binarizing the gradient amplitudes to determine the binarized gradient amplitudes; and finally, generating the edge mask image based on the binarized gradient amplitudes.

[0025] In conjunction with the first aspect, in an optional implementation, the expressions of the horizontal gradient and the vertical gradient are:

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] in, represents the horizontal gradient, represents the vertical gradient, represents the binary mask image, represents the Sobel operator in the horizontal direction, represents the Sobel operator in the vertical direction, represents the convolution operation, Represents the horizontal coordinate of the pixel point in the binary mask image, Represents the vertical coordinate of the pixel in the binary mask image. The expression of the gradient amplitude is:

[0031] ;

[0032] in, Represents the gradient amplitude. The expression of the gradient amplitude after binarization is:

[0033] ;

[0034] in, represents the gradient amplitude after binarization, Represents standard deviation.

[0035] In a second aspect, the present invention provides a moving target segmentation system based on satellite video, comprising: an acquisition module, a shape prior extraction module, and a moving target segmentation module. The acquisition module can be used to determine a reference frame image based on a query frame image to be segmented in a satellite video, and to determine a corresponding true mask image based on the reference frame image; the reference frame image is a frame image in a satellite video that has been segmented, and the true mask image is used to characterize the position and shape of the moving target in the reference frame image. The shape prior extraction module can be used to perform shape prior extraction based on the true mask image to determine an edge mask image, which is used to characterize the edge shape of the moving target. The moving target segmentation module can be used to perform feature extraction, spatiotemporal semantic relationship modeling, and semantic affinity constraint processing on the query frame image, the reference frame image, the true mask image, and the edge mask image using a moving target segmentation model, and output a target mask image, which is used to characterize the moving target segmentation result.

[0036] In a third aspect, the present invention provides an electronic device comprising: a memory, one or more processors; the memory is coupled to the processor; wherein the memory stores computer program code, the computer program code comprises computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method provided in the first aspect and any optional implementation thereof.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium comprising computer instructions, which, when executed on an electronic device, enables the electronic device to execute the method provided in the first aspect and any optional implementation thereof.

[0038] It can be understood that the beneficial effects that can be achieved by the satellite video-based motion target segmentation system provided by the second aspect, the electronic device of the third aspect, and the computer-readable storage medium of the fourth aspect can be referred to the beneficial effects of the first aspect and any of its optional implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the principle of the moving target segmentation method based on satellite video provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of a flow chart of a moving target segmentation method based on satellite video provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of the moving target segmentation model provided in an embodiment of the present application;

[0042] Figure 4 Schematic diagram of experimental data samples provided in the embodiments of this application;

[0043] Figure 5 A comparison chart of the moving target segmentation results of the ablation experiment provided in the embodiment of the present application;

[0044] Figure 6 A schematic diagram of the structure of a moving target segmentation system based on satellite video provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following embodiments are described in detail, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following embodiments are not intended to represent all possible implementations consistent with the present application. They are merely examples of systems and methods consistent with certain aspects of the present application, as detailed in the claims.

[0046] Video satellites are a new type of Earth observation satellite. Compared to traditional Earth observation satellites, video satellites can continuously observe a specific area, thereby acquiring remote sensing images with high spatial and spectral resolution, as well as high temporal resolution. This improves the timeliness of traditional remote sensing applications, such as disaster monitoring, ocean monitoring, and ecosystem disturbance monitoring. Furthermore, they enable applications that are difficult to achieve with traditional remote sensing, such as traffic monitoring.

[0047] Intelligent processing and analysis of satellite video can automatically extract and analyze information from scenes of interest, providing crucial technical support for these practical applications and playing a significant role in them. Satellite video object detection, segmentation, and tracking are typical tasks. Among them, satellite video object segmentation enables fine-grained segmentation of objects of interest in time-continuous scenes. Compared to detection and tracking, object segmentation not only locates the object but also provides additional, detailed descriptive information, enabling more sophisticated analysis and applications.

[0048] Unlike camera imaging in natural scenes, satellite video imaging is achieved by looking down at the Earth from high altitude. Therefore, the resolution of satellite video data is typically around 1 meter, lower than that of general video and high-resolution remote sensing data. In satellite video, typical remote sensing targets, such as vehicles, ships, and aircraft, typically have small appearances, mundane shapes and textures, and low contrast. Furthermore, satellite video has a long imaging range and a high frame rate. In satellite video, targets change little between frames, while the background changes slowly, resulting in significant spatiotemporal redundancy. Satellite video images depict large-scale scenes, where objects of interest occupy a very small proportion, and many details are not noticeable. Furthermore, satellite video imaging has a single perspective, and the background changes slowly, so the perspective and background do not change significantly, leading to information redundancy. This is especially true for important rigid targets such as aircraft and ships, which are subject to a single perspective and do not deform. Furthermore, temporal information is a unique and important property of satellite video data. In video scenarios, humans are often interested in objects that dynamically change over time, such as in target tracking and real-time monitoring tasks. Therefore, in areas where rigid targets gather and dock, such as airports and ports, many targets have very similar appearances, and most of the static targets in the background are not the dynamic targets we are really concerned about.

[0049] For satellite video, methods for moving target segmentation are mainly divided into two categories: unsupervised and semi-supervised. Traditional unsupervised satellite video target segmentation methods mainly utilize the idea of foreground extraction. Commonly used moving foreground detection and extraction methods include optical flow models, background subtraction models, frame difference models, and visual background extraction models. However, these traditional foreground extraction models have high requirements for video quality and assume that the background is static. Due to the movement of the satellite video shooting platform and changes in viewing angle, the background often undergoes slow motion changes. As a result, these foreground extraction models often extract many background shadows and false positives. In addition, the above algorithms require a lot of manual intervention and have a single feature expression, which makes the processing process more complicated.

[0050] With the rapid development of deep learning and computer technology, a range of methods based on convolutional neural networks (CNNs) have been widely used for moving object segmentation in satellite video, thanks to their powerful feature representation and learning capabilities. These methods primarily rely on varying degrees of prior information as input. Some algorithms process each video frame independently, without considering the temporal information between frames. For example, the OSVOS (One-Shot Video Object Segmentation) method utilizes an offline-trained fully convolutional network and performs online training on images to achieve moving object segmentation in single-frame videos. However, since video frames are segmented only individually, without utilizing the temporal information of the video, the moving object segmentation results are poor.

[0051] In related technologies, a method for segmenting moving targets in satellite videos is developed by performing frame-by-frame optical flow estimation on satellite videos to learn and utilize the appearance and motion information of the target. Although this method has certain advantages in processing the combination of appearance and motion information, it is highly dependent on motion information and relies on optical flow estimation to extract motion features. The optical flow calculation may not be accurate enough for some targets that move rapidly or in complex backgrounds. In particular, in low-quality videos or when the background changes dynamically, the optical flow estimation is easily affected by noise and errors. At the same time, the dual-stream network and the multi-scale progressive fusion module need to process features from multiple stages, which increases the computational overhead and may lead to high computational complexity and memory usage of the model, limiting its applicability in real-time applications. For long video sequences, the frame-by-frame processing of the model may face a high temporal dependency problem. As the length of the video increases, the transmission of frame-by-frame information and the accumulation of semantic information may lead to a decrease in the model's ability to model temporal information, especially for complex scenes and targets that change over a long period of time.

[0052] It can be seen that the relevant methods for moving target segmentation based on satellite videos have problems such as complex calculation process, insufficient refinement of target segmentation edges, and insufficient mining and utilization of temporal information, which makes it impossible to achieve moving target segmentation efficiently and accurately.

[0053] In order to solve the above problems, the present invention provides a method and system for segmenting moving objects based on satellite video. Figure 1 The schematic diagram of the principle of the moving target segmentation method based on satellite video provided in the embodiment of the present application is as follows: Figure 1As shown in the figure, features are first extracted from the query frame image, reference frame image, ground truth mask image, and edge mask image. Then, a moving target segmentation model is used to model spatiotemporal semantic relationships based on the extracted features, enabling efficient interactive learning of semantic information within and between satellite video frames. Finally, the feature results enhanced based on spatiotemporal relationships are used to predict the target mask, outputting a target mask image to segment the moving target in the query frame image.

[0054] This method targets rigid moving targets, fully utilizing their non-deformable properties and employing a shape prior fusion method to significantly enhance edge feature representation. Furthermore, to address the problem of stationary targets, which, while not defined as pure background, interfere with the semantic learning of the moving target segmentation model, this method employs foreground-background semantic affinity constraints to enhance the robustness and accuracy of semantic learning within the moving target segmentation model. By performing feature extraction, spatiotemporal semantic relationship modeling, and semantic affinity constraint processing within the moving target segmentation model, it is possible to achieve precise segmentation of moving targets in satellite video and effectively adapt to complex backgrounds and illumination changes, thereby effectively improving the efficiency and accuracy of moving target segmentation in satellite video.

[0055] The following describes the solution provided by the embodiments of the present application in conjunction with the accompanying drawings.

[0056] Specifically, Figure 2 A schematic diagram of a moving target segmentation method based on satellite video provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the moving target segmentation method based on satellite video provided in the embodiment of the present application includes the following steps S101-S104:

[0057] S101 : determining a reference frame image based on a query frame image to be segmented in a satellite video, and determining a corresponding true mask image according to the reference frame image.

[0058] In this embodiment of the present application, the query frame image is a video frame image from the satellite video (sequence) to be processed, for which moving object segmentation is required. Based on historical information and reference data (i.e., the reference frame image and the true mask image), the moving object region in the query frame image can be predicted to achieve moving object segmentation.

[0059] Exemplarily, moving targets may include wide-body aircraft, narrow-body aircraft, rear-engine aircraft, four-engine aircraft, business jets, speedboats, yachts, cruise ships, cargo ships, warships, other ships, large cars, small cars, and trains.

[0060] The reference frame image is a segmented frame image in the satellite video. Specifically, the reference frame image and the query frame image are in the same satellite video. The reference frame image is a known frame image for which moving objects have been accurately segmented. In other words, the reference frame image provides historical moving object segmentation information and prior knowledge. In this way, the reference frame image can accurately provide moving object appearance information for moving object segmentation.

[0061] The real mask image corresponding to the reference frame image can be used to characterize the position and shape of the moving target in the reference frame image. The real mask image can be used to supervise the network moving target segmentation model to learn the spatial distribution of the moving target.

[0062] In one implementation, based on the query frame image, a reference frame image and a true mask image corresponding to the reference frame image can be determined from the satellite video using historical segmentation results and prior knowledge.

[0063] S102: Perform shape priori extraction based on the real mask image to determine an edge mask image.

[0064] Furthermore, in embodiments of the present application, shape prior extraction can be performed based on the real mask image to determine an edge mask image. Specifically, this edge mask image can be used to characterize the edge shape of a moving target, that is, to provide a priori features of the moving target. Thus, the edge mask image can enhance the boundary information of the moving target, avoiding unclear boundaries caused by low contrast between the moving target and the background or motion blur, thereby improving the accuracy of moving target segmentation.

[0065] In some embodiments, the above S102 may specifically include the following steps S1021-S1023:

[0066] S1021 : Preprocess the real mask image to obtain a preprocessed real mask image.

[0067] Specifically, the real mask image may be preprocessed by normalization processing, noise removal, etc., to improve the consistency and integrity of the preprocessed real mask image.

[0068] S1022 : Perform mask binarization processing on the pre-processed real mask image to obtain a binary mask image.

[0069] Specifically, the target area can be extracted from the mask annotation of multiple categories of moving targets in the pre-processed real mask image, and the target area is set to a value of 1 and the background is set to a value of 0 to generate a binary mask image. For example, based on a preset threshold T, a binary mask can be calculated to determine the binary mask image:

[0070] ;

[0071] in, represents the binary mask image, Represents the horizontal coordinate of the pixel point in the binary mask image, Represents the vertical coordinate of the pixel in the binary mask image, Indicates the preset grayscale threshold, Represents pixel points The pixel value of .

[0072] S1023 : Perform edge detection on the binary mask image to obtain an edge mask image.

[0073] Finally, edge detection can be performed based on the binary mask image obtained in S1022 to extract the moving target boundary and generate a highly continuous and refined target contour boundary, that is, to obtain an edge mask image.

[0074] In some embodiments, a gradient calculation method may be used for edge detection. Specifically, S1023 may include the following steps S10231-S10234:

[0075] S10231. Determine the horizontal gradient and vertical gradient in the image according to the binary mask image.

[0076] Specifically, the Sobel operator can be used to calculate the horizontal gradient and the vertical gradient. For example, the expressions of the horizontal gradient and the vertical gradient are:

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] in, represents the horizontal gradient, represents the vertical gradient, represents the binary mask image, represents the Sobel operator in the horizontal direction, represents the Sobel operator in the vertical direction, Represents a convolution operation.

[0082] S10232. Determine the gradient amplitude based on the horizontal gradient and the vertical gradient.

[0083] For example, the expression of the gradient amplitude is:

[0084] ;

[0085] in, Indicates the gradient magnitude. The intensity of the pixel value change in the binary mask image can be characterized. The area with large amplitude in the binary mask image usually corresponds to the boundary of the moving target. In order to determine the direction of the moving target boundary, the gradient direction can also be calculated. For example, the gradient direction The expression is:

[0086] .

[0087] S10233. Binarize the gradient amplitude to determine the binarized gradient amplitude.

[0088] Furthermore, the gradient magnitude can be binarized to retain the boundary points and remove noise. For example, the expression of the gradient magnitude after binarization is:

[0089] ;

[0090] in, represents the gradient amplitude after binarization, Represents standard deviation.

[0091] S10234. Generate an edge mask image according to the binarized gradient amplitude.

[0092] Finally, according to the binarized gradient amplitude determined in S10233, an edge mask image can be accurately generated to provide enhanced boundary information of the moving target.

[0093] S103 , performing feature extraction, spatiotemporal semantic relationship modeling, and semantic affinity constraint processing on the query frame image, the reference frame image, the true mask image, and the edge mask image through the motion target segmentation model, and outputting a target mask image.

[0094] In an embodiment of the present application, by fusing a feature learning network model with shape priors, i.e., a motion target segmentation model, a target mask image can be determined based on the query frame image, the reference frame image, the true mask image, and the edge mask image. The target mask image includes the mask of the segmented motion target. The target mask image is used to characterize the motion target segmentation result, thereby achieving accurate segmentation of the motion target.

[0095] Specifically, the moving object segmentation model extracts features from the query frame image, reference frame image, true mask image, and edge mask image. This fusion of the shape prior features of the true mask image and edge mask image improves the refinement of moving object segmentation edges, thereby enhancing moving object segmentation accuracy and robustness to complex background interference.

[0096] The extracted features can then be used to model spatiotemporal semantic relationships. This allows for efficient interaction of semantic information within and between frames, improving the accuracy of moving target segmentation. This approach not only leverages the temporal characteristics of satellite video to dynamically model moving targets, but also accurately captures their semantic features within static frames. This effectively mitigates the limitations of insufficient spatiotemporal information utilization and further improves the applicability of moving target segmentation in satellite videos with complex backgrounds and small-sized targets.

[0097] Furthermore, semantic affinity constraints are applied to the modeled features, and the target mask image is output. This allows for foreground and background semantic affinity constraints to be implemented, explicitly modeling the semantic relationship between stationary and moving objects. This resolves the problem of stationary objects interfering with semantic learning, thereby improving the robustness and generalization of semantic learning.

[0098] The moving target segmentation method based on satellite video provided by the embodiment of the present application significantly enhances the edge feature expression capability through the shape prior fusion method. At the same time, in order to address the interference of stationary targets on the semantic learning of the moving target segmentation model, the method adopts foreground and background semantic affinity constraints to improve the robustness and accuracy of the semantic learning of the moving target segmentation model. In this way, by performing feature extraction, spatiotemporal semantic relationship modeling and semantic affinity constraint processing through the moving target segmentation model, it is possible to achieve accurate segmentation of moving targets in satellite videos and effectively adapt to complex backgrounds and lighting changes, thereby effectively improving the efficiency and accuracy of moving target segmentation in satellite videos.

[0099] In some embodiments, Figure 3 This is a schematic diagram of the structure of the moving target segmentation model provided in the embodiment of the present application, such as Figure 3 As shown, the moving target segmentation model 300 includes: a feature extraction module 310 , a spatiotemporal semantic relationship modeling module 320 and a semantic affinity constraint segmentation decoding head module 330 .

[0100] The feature extraction module 310 can be configured to extract features representing the spatial information and edge characteristics of the object from the query frame image, the reference frame image, the true mask image, and the edge mask image, respectively, to obtain corresponding first, second, third, and fourth extracted features. The first, second, third, and fourth extracted features are then spliced together to determine a target spliced feature.

[0101] Specifically, the feature extraction module 310 can extract features based on the query frame image, the reference frame image, the true mask image, and the edge mask image through multiple convolutional layers. The extracted features represent the spatial information and edge characteristics of the object, namely the first extracted features, the second extracted features, the third extracted features, and the fourth extracted features. Furthermore, the extracted features can be spliced to fuse the representation information from different inputs.

[0102] Exemplarily, the feature extraction module 310 can adopt a three-path input layer structure. Among them, the first path input layer can adopt a regular (for example: 3X3) convolution layer to extract features from the query frame image. The second path input layer can adopt three convolution layers, which are respectively used to extract the reference frame image, the foreground of the true mask image and the background of the true mask image, and then add the output features of the three convolution layers as the final output features. The third path input layer can adopt a convolution layer to extract features from the edge mask image. Then, feature splicing is performed to obtain splicing features. Furthermore, the first four stages of the ResNet network can be used as feature extractors to extract features from the splicing features again. Finally, the input query frame image, reference frame image, true mask image and edge mask image are mapped to the feature map ,in Represents height, width and number of channels respectively, is the number of reference frame images.

[0103] Next, the feature map can be transformed into The number of spatial channels from Reduced to (For example: 256) to perform downsampling, realize lightweight processing of feature maps, and generate new feature maps Then, the new feature map can be The number of spatiotemporal channels is smoothed to one dimension, that is, from two dimensions to one dimension, so as to adjust the shape and size of the feature map to adapt to the input requirements of the Transformer and obtain the final target splicing feature. .

[0104] The spatiotemporal semantic relationship modeling module 320 can be used to perform position coding addition, spatiotemporal semantic relationship modeling coding, and target prediction decoding according to target splicing features, and determine coding features and decoding features.

[0105] In an embodiment of the present application, the spatiotemporal semantic relationship modeling module 320 employs position encoding. By incorporating spatiotemporal position information, the encoder can effectively capture the spatiotemporal dependencies between pixels in the input frame through spatiotemporal semantic relationship modeling encoding. This allows the moving target segmentation model 300 to model the dynamic changes between objects within multiple time steps, thereby improving the accuracy of target detection and tracking. Furthermore, the encoder can also learn and establish the spatial-temporal correspondence between the query frame image and the reference frame image through spatiotemporal semantic relationship modeling encoding. In this way, the moving target segmentation model 300 can learn and understand the relative changes of the moving target in different frame images and effectively model the structural features of the moving target in a specific frame. The decoder can predict the spatial position information of the moving target in each frame image through target prediction decoding, thereby enhancing the spatiotemporal reasoning capability of the moving target segmentation model 300. In this way, not only can the transformation of the moving target between different time steps be accurately captured, but by learning a robust representation of the target object, the moving target segmentation model 300 can still respond quickly and maintain efficient performance when faced with confusion problems of similar objects. This mechanism is particularly suitable for complex moving target tracking scenarios, significantly improving the processing speed and accuracy of the moving target segmentation model 300 in dynamic environments. Therefore, the spatiotemporal semantic relationship modeling module 320 combines spatiotemporal modeling with position encoding, enabling the moving target segmentation model 300 to effectively process the spatiotemporal information of moving targets, addressing the diversity and uncertainty issues in moving target segmentation and thereby improving the accuracy of moving target segmentation.

[0106] Specifically, in some embodiments, Figure 3 As shown, the spatiotemporal semantic relationship modeling module 320 specifically includes: a position encoding module 321, a Transformer encoder 322 and a Transformer decoder 323.

[0107] The position coding module 321 may be used to add position coding to the target splicing feature to obtain a position coding splicing feature.

[0108] In this embodiment, since the self-attention mechanism is not capable of capturing sequence position information, positional encoding (PE) can be introduced to provide position information.

[0109] In one implementation, the position coding module 321 can use a sinusoidal position coding method to add position coding to the target splicing feature. The sinusoidal position coding is based on a fixed trigonometric function pattern, so that the coding of different positions is unique and can maintain a certain inductive ability under different sequence lengths of the target splicing feature. For example, the expression of the position coding splicing feature can be:

[0110] ;

[0111] ;

[0112] ;

[0113] in, represents the positional encoding splicing feature, represents the target splicing feature, represents the sinusoidal position encoding, Represents the two-dimensional position index of the element in the target splicing feature, represents the index of the feature dimension (sine for even indices and cosine for odd indices), Represents the dimension of the positional encoding (i.e., the input feature dimension of the Transformer encoder).

[0114] The Transformer encoder 322 can be used to concatenate features according to positional encoding, perform spatiotemporal semantic relationship modeling encoding through a multi-head self-attention module and a fully connected feedforward neural network, and determine and output encoding features.

[0115] In this embodiment, the Transformer encoder 322 can learn the semantic information within and between frames to perform spatiotemporal semantic relationship modeling and encoding to form a high-dimensional feature representation, that is, determine and output the encoding features.

[0116] Specifically, the Transformer encoder 322 can be used to encode the positional concatenation features Model the spatiotemporal relationship and output the encoded features . Encoded features after encoding After global feature extraction using a multi-head self-attention module and a fully connected feedforward neural network, the motion target segmentation model 300 is able to capture pixel-level semantic relationships within frames and temporal associations between frames.

[0117] Among them, the encoding feature It has stronger global representation ability. Specifically: in the spatial dimension, the encoding features Through the multi-head self-attention mechanism, the position encoding splicing features are learned The long-distance dependency between pixels in the image strengthens the structural integrity of the foreground object and reduces the impact of local noise. In the temporal dimension, the encoding features By integrating the motion information between frames, the position and shape changes of the moving target can be modeled, and the spatiotemporal consistency of the dynamic moving target can be improved. After multiple layers of nonlinear transformation, relevant information can be aggregated more closely, making the semantic representation clearer and more robust.

[0118] For example, the Transformer encoder 322 can consist of eight encoding layers, each of which includes standard architectural modules: a multi-head self-attention module and a fully connected feedforward network. By employing eight different multi-head self-attention modules, the encoder can capture the spatiotemporal dependencies between pixels in the input sequence in parallel across multiple subspaces, thereby enhancing its understanding of the object's motion patterns.

[0119] The Transformer decoder 323 can be used to perform target prediction decoding based on the encoding features through a multi-head self-attention module, a multi-head cross-attention module and a fully connected feedforward neural network to determine and output the decoding features.

[0120] In this embodiment, the spatial position information of the moving target in each frame image (such as the query frame image) can be predicted by the Transformer decoder 323. The Transformer decoder 323 receives the encoding features output by the Transformer encoder 322. Determine and output decoding features through multi-head self-attention module, multi-head cross attention module and fully connected feedforward neural network , decoding features It can be used to generate target mask images.

[0121] During the training phase of the moving object segmentation model 300, the Transformer decoder 323 can also be used to train the moving object segmentation model 300 according to the encoding features. And training target query Determine and output decoding features using multi-head self-attention modules, multi-head cross-attention modules, and fully connected feedforward neural networks Among them, the training target query The feature query vector representing the moving target to be predicted can be used to extract the corresponding moving target features.

[0122] Specifically, the decoding features output by Transformer decoder 323 are It is a high-dimensional representation of the spatial position information and semantic features of the moving target in each frame of the image. It combines the global spatiotemporal information output by the encoder and the target query trained during the training phase. The extracted target-specific information, after multi-layer decoding, can accurately depict the spatial contours of the moving target and its temporal changing trends.

[0123] Among them, the decoding feature It has the following features: Target area prediction: decoding features The final target mask image can be generated through upsampling and convolution layers to achieve pixel-level moving target area segmentation. Combined with context information, it can be used to optimize the edge details of moving targets and improve the accuracy of moving target segmentation results. Temporal consistency enhancement: decoding features It inherits the temporal and spatial correlation modeling capabilities of the Transformer encoder, ensuring that the moving target segmentation results remain consistent between sequence frames and reducing the impact of timing jitter and drift.

[0124] Exemplarily, the Transformer decoder 323 can be composed of 8 decoder layers, each of which includes a multi-head self-attention module, a multi-head cross-attention module, and a fully connected feedforward network. The multi-head cross-attention module can effectively deeply fuse the target features obtained in the Transforme encoder stage with the query information of the Transforme decoder, further optimizing the spatiotemporal feature modeling of moving targets. In this way, the Transformer decoder 323 can better integrate and utilize the feature information extracted by the Transforme encoder when processing complex scenes, thereby improving the positioning accuracy of moving targets.

[0125] As can be seen, the Transformer decoder 323, through the interaction and optimization of multiple decoding layers, can ensure efficient prediction and stable output of the spatial position information of moving targets. The Transformer's spatiotemporal relationship modeling capabilities and the Transformer decoder's complex multi-head cross-attention mechanism can effectively address issues in the moving target segmentation task, including multi-target confusion, target occlusion, and motion trajectory changes, thereby significantly improving the accuracy of the moving target segmentation model 300 in dynamic environments.

[0126] The semantic affinity constraint segmentation decoding head module 330 can be used to perform mask prediction based on the encoding features and the decoding features to determine the target mask image.

[0127] In this embodiment, semantically consistent robust representation learning is achieved through the semantic affinity constraint segmentation decoding head module 330. During moving object segmentation, ground truth annotations can provide information about the category to which each pixel belongs. However, the moving object segmentation model 300 struggles to learn complete contextual information from a single pixel-level feature. This is particularly true in Earth observation satellite video scenarios, where objects are small and the ratio of foreground and background pixels is severely unbalanced, further increasing the learning difficulty of the moving object segmentation model 300.

[0128] In this context, stationary background targets (such as ground objects or environmental features) usually have similar appearance and category as moving targets. Since stationary targets do not move and usually belong to the same category or model as the target of interest, they are easily misjudged as background. In the case where these stationary targets are misclassified as background, it will cause significant deviations in the semantic information learning of the moving targets, seriously affecting the accurate segmentation and recognition of the moving targets. The embodiment of the present application adopts a semantic affinity constraint mechanism, that is, the learning process of the moving target segmentation model 300 is explicitly constrained by the semantic affinity constraint segmentation decoding head module 330, so that the moving target segmentation model 300 can more accurately distinguish between moving targets and stationary targets, and optimize the semantic consistency between the background and the foreground.

[0129] The semantic affinity-constrained segmentation decoding head module 330 explicitly regulates the training of the moving object segmentation model 300, strengthening the semantic distinction between the target and the background. This reduces the risk of misclassifying stationary objects as background, thereby improving the moving object segmentation performance of the moving object segmentation model 300 in complex scenes. The semantic affinity-constrained segmentation decoding head module 330 can effectively resolve the semantic confusion problem in moving object segmentation, especially when the appearance of stationary and moving objects is similar. It can effectively improve the robustness and accuracy of the moving object semantic learning of the moving object segmentation model 300.

[0130] In an embodiment of the present application, the semantic affinity constraint segmentation decoding head module 330 is composed of a dual-branch parallel processing architecture, including a target mask prediction branch (i.e., a target mask prediction module) and a semantic affinity prediction branch (i.e., a semantic affinity prediction module), and realizes the segmentation of moving targets in satellite videos through multi-scale feature fusion and cross-modal attention mechanism.

[0131] Specifically, in some embodiments, Figure 3 As shown, the semantic affinity constraint segmentation decoding head module 330 includes: a target affinity module 331 and a target mask prediction module 332.

[0132] Among them, the target affinity module 331 can be used to determine and output a self-attention feature map based on the decoding features and the encoding features.

[0133] Specifically, the target affinity module 331 can use the encoded feature as the query Q and the decoded feature as the key K and the value V, and determine and output the self-attention feature map based on the scaled dot-product attention (SDPA) mechanism.

[0134] The target mask prediction module 332 can be used to determine the target mask image through upsampling and convolution processing based on the self-attention feature map and the encoded features.

[0135] Exemplarily, the target mask prediction module 332 may include: multiple (for example: 3) feature coordination blocks, 2 convolutional layers (1×1, or 3×1) and Softmax activation to achieve upsampling and convolution processing of the self-attention feature map and the encoded features to determine the target mask image.

[0136] Among them, feature collaboration can adopt a three-level cascade processing unit structure, which can fuse multi-scale features through jump connections, reduce computational complexity with depthwise separable convolution, and generate 256-channel intermediate features step by step.

[0137] In some embodiments, as Figure 3 As shown, the semantic affinity constraint segmentation decoding head module 330 further includes: a semantic affinity prediction module 333 .

[0138] The semantic affinity prediction module 333 can be used to generate a semantic affinity map based on the self-attention feature map through convolution, batch normalization, and sigmoid activation. The semantic affinity map is a heat map used to represent the semantic relevance between foreground and background. The semantic affinity map is used to train the moving object segmentation model.

[0139] In an embodiment of the present application, during the training process of the motion target segmentation model 300, a semantic affinity graph can be generated through the semantic affinity prediction module 333 to provide strong supervision of the semantic relationship between pixel points for the motion target segmentation model 300, thereby optimizing the accuracy and robustness of the motion target segmentation model 300 in segmenting motion targets.

[0140] Exemplarily, the semantic affinity prediction module 333 may include: a feature adaptation module and a prediction layer. The feature adaptation module may adopt a fully separable convolution decomposition based on the self-attention feature map. Core and cascade operation and generate adaptive features through channel compression The prediction layer can include: 1×1 convolution layer, batch normalization layer and Sigmoid activation. The prediction layer can be used to adapt the features. Generate and output a semantic affinity graph.

[0141] In some embodiments, to construct an ideal semantic affinity graph, the sample frame images of the input satellite video first have corresponding true value labels The sample frame image is processed by the feature extraction module and the spatiotemporal semantic relationship modeling module to extract high-dimensional feature representations. Subsequently, these high-dimensional feature representations can be input into the multi-layer perceptron (MLP) and segmentation head, and the final output size is Segmentation feature map.

[0142] In order to be consistent with the segmentation feature map, the true value label Downsampled to the same resolution as the segmentation feature map, denoted as Then, Perform one-hot encoding and Convert to a dimension of tensor, where is the total number of segmentation categories. This dimension is The tensor is reshaped and reorganized into The matrix form of Represents the total number of pixels in the segmentation feature map.

[0143] In the process of constructing the semantic affinity graph, by calculating ( is the label tensor after adjusting the shape), you can get a The symmetric matrix A is a symmetric matrix. This matrix A can intuitively represent the category affinity relationship between pixels, that is, the ideal semantic affinity graph. Through this semantic affinity graph, the moving object segmentation model can be provided with strong supervision of the semantic relationship between pixels, thereby optimizing the accuracy and robustness of moving object segmentation.

[0144] In some embodiments, the loss function for training the moving object segmentation model 300 is expressed as:

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] in, represents the loss function of the moving target segmentation model, represents the classification loss of pixels in the frame image, represents the target mask prediction loss, Indicates affinity loss, represents the set of pixels in the target mask image, Indicates the target of movement The predicted mask image, Indicates the target of movement The true mask image of Indicates the number of moving targets, represents the size of the semantic affinity graph, Represents the feature points of the semantic affinity graph, represents the predicted probability of the semantic affinity graph, represents the true value of the target probability distribution, Indicates the sports goals, Indicates the The target mask image corresponding to the moving target, Indicates the target mask image at pixel point The value at Indicates the true value mask image at pixel point The value at .

[0150] In some embodiments, a verification analysis is performed to verify the performance and effectiveness of the moving target segmentation method based on satellite video provided in the above embodiments of the present application (hereinafter referred to as: the technical solution of the present invention).

[0151] Experimental Dataset: We selected the dataset from the Satellite Video Multiple Tasks Benchmark (SAT-MTB) dataset as our experimental dataset. The SAT-MTB dataset supports multiple tasks, including target detection, target tracking, and target segmentation. The experimental dataset includes moving targets. Specifically, the experimental dataset consists of 30 videos with a frame rate of 10 frames per second, an average duration of approximately 22 seconds, and a total of 5,226 frames.

[0152] For example, Figure 4 The following is a schematic diagram of an experimental data sample provided in the embodiment of this application, as shown in FIG. Figure 4 (a) is a schematic diagram of an experimental data sample where the moving target is an airplane. Figure 4 (b) is a schematic diagram of an experimental data sample where the moving target is a ship.

[0153] Evaluation method: This technical solution uses J-Mean, F-Mean, and J&F Mean evaluation indicators to evaluate the similarity between the moving target segmentation results and the true labels.

[0154] Specifically, J-Mean refers to the average value of the intersection over union (IoU) of the segmentation result and the true label, with a value range of 0-1. J-Mean calculates the IoU of each sample and averages the IoU of all samples. It can be used to measure the regional overlap between the moving target segmentation result and the true label.

[0155] F-Mean is the average value of F1-score based on the segmentation boundary accuracy. It can be used to evaluate the matching degree between the moving target segmentation result and the true label boundary. The value range is 0-1.

[0156] The specific calculation method is as follows:

[0157]

[0158] in, Indicates the proportion of pixels captured by the moving target segmentation model in the real moving target. Indicates that the pixels predicted by the moving target segmentation model are moving targets, which are consistent with the true labels. Indicates the moving target pixels that are missed by the moving target segmentation model, i.e., missed detection, Indicates the proportion of moving target pixels predicted by the moving target segmentation model that are actually moving targets, Indicates that the moving target segmentation model incorrectly predicts background pixels as moving targets, i.e., false positives, It indicates the result of considering the overall balance between Recall and Precision.

[0159] J&F Mean is the arithmetic mean of J-Mean and F-Mean, and can be used to comprehensively evaluate the performance of segmentation regions and segmentation boundaries. It comprehensively considers the performance of region overlap (J-Mean) and boundary matching (F-Mean) to provide a more comprehensive segmentation quality evaluation indicator. The specific calculation method is as follows:

[0160] ;

[0161] in, Indicates J&F Mean, represents J-Mean, Indicates F-Mean.

[0162] The technical solution of the present invention is compared with the related art 1 RVOS method and the related art 2 OSVOS method in the accuracy of satellite video moving target segmentation. The experimental results are shown in Table 1.

[0163] Table 1 Comparative experimental results of satellite video moving target segmentation accuracy

[0164]

[0165] The final J&F Mean values of different methods are shown in Table 2.

[0166] Table 2 Comparison results of J&F mean values

[0167]

[0168] The technical solution of the present invention is applied to the SAT-MTB dataset. is 0.409, is 0.554, The value is 0.471, outperforming both existing technical solutions in terms of accuracy. This shows that by integrating the salient features of moving ships, higher-quality pseudo-label samples can be obtained. The noise-resistant target detection network, during the training and learning process of the moving target segmentation model, can resist the interference caused by boundary noise in the pseudo-label samples of ships, achieving more accurate and detailed feature descriptions of moving ships. Finally, candidate region consistency regression further improves target positioning accuracy.

[0169] Moreover, the difference between the technical solution of the present invention and the related art RVOS method is that the technical solution of the present invention adopts an end-to-end approach, and the significant fusion feature construction of the satellite video sequence frames and the motion foreground extraction are only used to obtain pseudo-label samples in the training phase. After the moving target segmentation model completes the network training, the moving target segmentation model can directly perform fast inference on the original satellite video to be tested, with higher inference efficiency. Therefore, a statistical comparison experiment on the efficiency of moving ship detection was conducted to compare the model inference time of the technical solution of the present invention, the related art RVOS method and the related art OSVOS method on the SAT-MTB test set, and quantitatively analyze the detection efficiency of moving ships in satellite videos. The experimental results show that the technical solution of the present invention achieves better moving ship detection results while taking less time to infer the model. The detection speed is not only significantly ahead of the related art RVOS method, but also better than the related art OSVOS method.

[0170] Furthermore, in some embodiments, an ablation experiment analysis was performed to verify the effect of each step in the technical solution of the present invention. By gradually adding and introducing spatiotemporal semantic affinity constraints (i.e., the above-mentioned S103), shape priors (i.e., the above-mentioned S102), and a combination of the two (i.e., the technical solution of the present invention), the impact of each step on the accuracy of satellite video moving target segmentation was analyzed. Figure 5 This is a comparison chart of the motion target segmentation results of the ablation experiment provided in the embodiment of this application, such as Figure 5 As shown in (1), it is a schematic diagram of the segmentation results obtained by using the baseline method (Baseline) to segment moving targets in satellite videos. Figure 5As shown in (2), it is a schematic diagram of the segmentation results obtained by using the benchmark method and spatiotemporal semantic affinity constraints to segment moving targets in satellite videos. Figure 5 As shown in (3), it is a schematic diagram of the segmentation results obtained by using the benchmark method and shape prior to segment the moving target in the satellite video. Figure 5 As shown in (4), it is a schematic diagram of the segmentation results obtained by segmenting the moving target in the satellite video using the benchmark method, spatiotemporal semantic affinity constraints and shape prior (i.e. the technical solution of the present invention). Figure 5 (5) is a schematic diagram of the ground truth labels of the segmentation results of moving targets in satellite videos. Table 3 shows the ablation experiment results of the segmentation of moving targets in satellite videos.

[0171] Table 3 Satellite video moving target segmentation ablation experiment results

[0172]

[0173] Table 3 shows the results of ablation experiments on moving object segmentation conducted on eight typical satellite video sequences (including those of ships and aircraft). The evaluation metrics used were J-Mean and F-Mean, which measure the overlap and boundary contour similarity between the segmented regions and the ground-truth annotations. Starting from a baseline approach, the ablation experiments introduced spatiotemporal semantic affinity constraints, shape priors, and a combined strategy of the two, comparing their performance across different object types and background scenarios. The results show that the introduction of affinity constraints improves the J-Mean values of the moving object segmentation model on most sequences, particularly in videos with small foreground objects and complex backgrounds, such as Air1 and Air2. This demonstrates that the affinity mechanism effectively enhances cross-frame semantic consistency and foreground recognition. The introduction of shape priors also achieves good results on objects with stable structures and clear boundaries, such as those in aircraft sequences. This is particularly evident in sequences like Air1, Air4, and Air5, where the moving object segmentation model not only accurately segments the foreground but also maintains smooth and consistent edges. When the spatiotemporal semantic affinity constraints and shape priors are jointly introduced, i.e., when the technical solution of the present invention is adopted, the moving object segmentation model achieves optimal J-Mean and F-Mean on most videos, validating the complementary enhancement of the technical solution in preserving moving object contours and aggregating semantic information. Overall, the technical solution of the present invention can more comprehensively improve the perception and segmentation quality of moving foreground objects in the moving object segmentation model, taking into account semantic consistency and structural constraints, and demonstrates good generalization and robustness for moving object segmentation.

[0174] The embodiment of the present application also provides a moving target segmentation system based on satellite video, specifically, Figure 6 This is a schematic diagram of the structure of the moving target segmentation system based on satellite video provided in an embodiment of the present application. Figure 6 As shown, the moving target segmentation system 600 based on satellite video includes: an acquisition module 601 , a shape prior extraction module 602 and a moving target segmentation module 603 .

[0175] Among them, the acquisition module 601 can be used to determine a reference frame image based on the query frame image to be segmented in the satellite video, and determine the corresponding true mask image based on the reference frame image; the reference frame image is a frame image in the satellite video that has been segmented by the target, and the true mask image is used to represent the position and shape of the moving target in the reference frame image.

[0176] The shape prior extraction module 602 may be used to perform shape prior extraction based on the real mask image to determine an edge mask image. The edge mask image is used to characterize the edge shape of the moving target.

[0177] The motion target segmentation module 603 can be used to perform feature extraction, spatiotemporal semantic relationship modeling and semantic affinity constraint processing on the query frame image, reference frame image, true mask image and edge mask image through the motion target segmentation model, and output a target mask image, which is used to represent the motion target segmentation result.

[0178] The satellite video-based moving target segmentation system provided by the embodiment of the present application significantly enhances the edge feature expression capability through the shape prior fusion method. At the same time, in order to address the interference of stationary targets on the semantic learning of the moving target segmentation model, the system adopts foreground and background semantic affinity constraints to improve the robustness and accuracy of the semantic learning of the moving target segmentation model. In this way, by performing feature extraction, spatiotemporal semantic relationship modeling and semantic affinity constraint processing through the moving target segmentation model, it is possible to achieve accurate segmentation of moving targets in satellite videos and effectively adapt to complex backgrounds and lighting changes, thereby effectively improving the efficiency and accuracy of moving target segmentation in satellite videos.

[0179] An embodiment of the present invention further provides an electronic device, which may include: a display screen, a memory, and one or more processors. The display screen, memory, and processor are coupled. The memory is configured to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform the various methods or steps described in the embodiment of the method for moving object segmentation based on satellite video. Of course, the electronic device includes, but is not limited to, the display screen, memory, and one or more processors described above.

[0180] An embodiment of the present invention further provides a computer-readable storage medium for storing computer instructions for executing the above-mentioned method for moving target segmentation based on satellite video.

[0181] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0182] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0183] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0184] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without expending creative work shall fall within the scope of protection of this application.

Claims

1. A moving target segmentation method based on satellite video, characterized in that: include: Determining a reference frame image based on a query frame image to be segmented in a satellite video, and determining a corresponding true mask image according to the reference frame image; The reference frame image is a target-segmented frame image in the satellite video, and the real mask image is used to represent the position and shape of the moving target in the reference frame image; Performing shape priori extraction based on the real mask image to determine an edge mask image, wherein the edge mask image is used to characterize the edge shape of the moving target; Performing feature extraction, spatiotemporal semantic relationship modeling, and semantic affinity constraint processing on the query frame image, the reference frame image, the true mask image, and the edge mask image through a moving target segmentation model, and outputting a target mask image, wherein the target mask image is used to represent a moving target segmentation result; The moving target segmentation model includes: a feature extraction module and a spatiotemporal semantic relationship modeling module; The feature extraction module is used to extract features for characterizing object spatial information and edge characteristics from the query frame image, the reference frame image, the true mask image, and the edge mask image, respectively, to obtain corresponding first extracted features, second extracted features, third extracted features, and fourth extracted features, and to splice the first extracted features, the second extracted features, the third extracted features, and the fourth extracted features to determine a target splicing feature; The spatiotemporal semantic relationship modeling module is used to perform position coding addition, spatiotemporal semantic relationship modeling coding and target prediction decoding according to the target splicing features, and determine coding features and decoding features; The spatiotemporal semantic relationship modeling module includes: a position encoding module, a Transformer encoder and a Transformer decoder; The position coding module is used to add position coding to the target splicing feature to obtain a position coding splicing feature; The Transformer encoder is used to perform spatiotemporal semantic relationship modeling encoding based on the position coding splicing features through a multi-head self-attention module and a fully connected feedforward neural network to determine and output the coding features; The Transformer decoder is used to perform target prediction decoding based on the encoding features through a multi-head self-attention module, a multi-head cross-attention module and a fully connected feedforward neural network to determine and output the decoding features.

2. The method according to claim 1, characterized in that The moving target segmentation model further includes: a semantic affinity constraint segmentation decoding head module; The semantic affinity constraint segmentation decoding head module is used to perform mask prediction based on the encoding features and the decoding features to determine the target mask image.

3. The method according to claim 2, characterized in that The semantic affinity constraint segmentation decoding head module includes: a target affinity module and a target mask prediction module; wherein, The target affinity module is configured to determine and output a self-attention feature map based on the decoding features and the encoding features; The target mask prediction module is used to determine the target mask image through upsampling and convolution processing based on the self-attention feature map and the encoding features.

4. The method according to claim 3, characterized in that The semantic affinity constraint segmentation decoding head module further includes: a semantic affinity prediction module; The semantic affinity prediction module is used to generate a semantic affinity graph based on the self-attention feature graph through convolution processing, batch normalization processing and Sigmoid activation processing; wherein the semantic affinity graph is a heat map used to characterize the semantic correlation between foreground and background, and the semantic affinity graph is used to train the moving target segmentation model.

5. The method according to claim 4, characterized in that The loss function of the moving target segmentation model is expressed as: ； ； ； ； in, represents the loss function of the moving target segmentation model, represents the classification loss of pixels in the frame image, represents the target mask prediction loss, Indicates affinity loss, represents the set of pixels in the target mask image, Indicates the target of movement The predicted mask image, Indicates the target of movement The true mask image of Indicates the number of moving targets, represents the size of the semantic affinity graph, Represents the feature points of the semantic affinity graph, represents the predicted probability of the semantic affinity graph, represents the true value of the target probability distribution, Indicates the sports goals, Indicates the The target mask image corresponding to the moving target, Indicates the target mask image at pixel point The value at Indicates the true value mask image at pixel point The value at .

6. The method according to claim 1, characterized in that The performing shape prior extraction according to the real mask image to determine the edge mask image includes: Preprocessing the real mask image to obtain a preprocessed real mask image; performing mask binarization processing on the pre-processed real mask image to obtain a binary mask image; Edge detection is performed on the binary mask image to obtain the edge mask image.

7. The method according to claim 6, characterized in that The performing edge detection on the binary mask image to obtain the edge mask image includes: Determining a horizontal gradient and a vertical gradient in an image according to the binary mask image; determining a gradient amplitude based on the horizontal gradient and the vertical gradient; performing binarization processing on the gradient amplitude to determine the binarized gradient amplitude; The edge mask image is generated according to the binarized gradient amplitude.

8. The method according to claim 7, characterized in that The expressions of the horizontal gradient and the vertical gradient are: ； ； ； ； in, represents the horizontal gradient, represents the vertical gradient, represents the binary mask image, represents the Sobel operator in the horizontal direction, represents the Sobel operator in the vertical direction, represents the convolution operation, Represents the horizontal coordinate of the pixel point in the binary mask image, Represents the vertical coordinate of the pixel in the binary mask image; The expression of the gradient amplitude is: ； in, represents the gradient amplitude; The expression of the gradient amplitude after binarization is: ； in, represents the gradient amplitude after binarization, Represents standard deviation.

9. A moving target segmentation system based on satellite video, characterized in that: include: Acquisition module, shape prior extraction module, moving target segmentation module; among them, The acquisition module is configured to determine a reference frame image based on a query frame image to be segmented in the satellite video, and to determine a corresponding true mask image based on the reference frame image; the reference frame image is a frame image in the satellite video that has been segmented, and the true mask image is used to represent the position and shape of the moving target in the reference frame image; The shape prior extraction module is used to perform shape prior extraction based on the real mask image to determine an edge mask image, where the edge mask image is used to characterize the edge shape of the moving target; The moving target segmentation module is used to perform feature extraction, spatiotemporal semantic relationship modeling, and semantic affinity constraint processing on the query frame image, the reference frame image, the true mask image, and the edge mask image through a moving target segmentation model, and output a target mask image, which is used to represent the moving target segmentation result; The moving target segmentation model includes: a feature extraction module and a spatiotemporal semantic relationship modeling module; The feature extraction module is used to extract features for characterizing object spatial information and edge characteristics from the query frame image, the reference frame image, the true mask image, and the edge mask image, respectively, to obtain corresponding first extracted features, second extracted features, third extracted features, and fourth extracted features, and to splice the first extracted features, the second extracted features, the third extracted features, and the fourth extracted features to determine a target splicing feature; The spatiotemporal semantic relationship modeling module is used to perform position coding addition, spatiotemporal semantic relationship modeling coding and target prediction decoding according to the target splicing features, and determine coding features and decoding features; The spatiotemporal semantic relationship modeling module includes: a position encoding module, a Transformer encoder and a Transformer decoder; The position coding module is used to add position coding to the target splicing feature to obtain a position coding splicing feature; The Transformer encoder is used to perform spatiotemporal semantic relationship modeling encoding based on the position coding splicing features through a multi-head self-attention module and a fully connected feedforward neural network to determine and output the coding features; The Transformer decoder is used to perform target prediction decoding based on the encoding features through a multi-head self-attention module, a multi-head cross-attention module and a fully connected feedforward neural network to determine and output the decoding features.

Images