Mobile instance segmentation method and system

Abstract

The application discloses a mobile instance segmentation method and system, and belongs to the field of pattern recognition. In order to solve the problem that the existing method has insufficient small target segmentation precision in extreme scenes such as low light and high speed motion due to image degradation, entanglement of event appearance and motion characteristics, event sparsity and solidification of modal division, the application proposes an innovative multi-modal fusion framework. The framework first adopts a double-decoupling encoder to simultaneously extract appearance and motion dual features from image and event streams; cross-modal feature alignment and fusion are realized by using adversarial training and reversible conversion; feature interaction is promoted by combining cross-attention; and finally, instance masks and motion states are output through task decoding. The application is suitable for fields such as automatic driving and intelligent monitoring, and has the advantages of high small target segmentation precision, strong fusion robustness and excellent performance in extreme scenes.

Description

Technical Field

[0001] This invention relates to the field of pattern recognition, and more specifically to a moving instance segmentation method and system. Background Technology

[0002] Moving instance segmentation (MIS) is a key task in the field of pattern recognition, aiming to identify, segment, and track multiple independently moving object instances from video sequences for pattern recognition. This technology has wide-ranging applications in autonomous driving, intelligent surveillance, robot navigation, and biological behavior analysis.

[0003] Traditional image sensor-based motion instance segmentation methods primarily rely on deep learning models to extract appearance and motion features from consecutive frames. These methods perform well under normal lighting and motion conditions, but in extreme scenarios such as low lighting, high dynamic range, and high-speed motion, the inherent technical limitations of image sensors lead to severe degradation of the captured signal (such as motion blur and reduced signal-to-noise ratio), rendering appearance-based feature extraction and motion estimation essentially ineffective.

[0004] To overcome the limitations of traditional image sensors, event cameras have attracted attention as a novel type of sensor. Event cameras asynchronously record the brightness changes (events) of each pixel, featuring microsecond-level latency, high dynamic range, and low power consumption, making them particularly adept at capturing fast motion. However, event data is inherently a sparse, asynchronous point cloud stream, lacking direct texture and appearance information, and its spatial resolution is typically lower than that of traditional cameras. This makes it difficult for event-based methods alone to generate dense and accurate object boundaries in moving instance segmentation tasks, especially for structurally complex objects or multiple instances with similar appearances.

[0005] Naturally, multimodal fusion has become a research hotspot. Existing multimodal fusion methods generally follow a paradigm of "modal function solidification". Figure 1 This is a schematic diagram of existing multimodal feature extraction technology. As shown in the figure, appearance and texture features are mainly extracted from image frames, while motion and temporal features are mainly extracted from event streams. These features are then fused at the feature layer or decision layer. Although this approach improves performance to some extent, this rigid division of labor—"images manage appearance, events manage motion"—has inherent drawbacks:

[0006] First, it fails to fully exploit the latent information within each modality. Successive frames in an image sequence contain rich motion cues (such as optical flow and motion blur), while the density and spatial distribution patterns of event flows implicitly reveal appearance information such as object contours and textures. Existing methods neglect the collaborative mining of complementary cues within these modalities.

[0007] Secondly, since event data is inherently sparse in small target regions, extracting only motion features from it would result in extremely limited feature dimensions to describe the target, leading to insufficient representational power. Under extreme conditions, when the image modality is severely degraded, the limited motion features provided by the event modality cannot effectively compensate for the lost appearance information, significantly reducing the fusion effect. Furthermore, event data is triggered by both target motion and surface attributes, resulting in a high degree of coupling between appearance and motion information. Figure 2 This is a feature similarity comparison chart between the image modality and the event modality. As shown in the figure, on the MouseSIS dataset, the curves display the variation of cosine similarity between appearance features and motion features extracted from the image encoder and event encoder, respectively. This curve indicates that, at different iterations, the similarity between the appearance and motion components of the features extracted from the event modality is consistently higher than that from the image modality. This directly confirms the feature entanglement problem in event data. The directly extracted "motion features" are actually mixed with appearance attributes, resulting in impure features.

[0008] Finally, image and event data originate from different physical imaging processes, and their feature distributions differ statistically. Existing fusion methods mostly focus on simple feature stitching or attention-based weighting, lacking systematic constraints on the deep distribution alignment and semantic consistency of cross-modal features. This leads to ambiguity in the fused features, affecting segmentation accuracy, especially when dealing with multiple instances that are similar in appearance and have overlapping motions.

[0009] In more fundamental research areas, work on video object segmentation (VOS) (such as semi-supervised methods like FEELVOS and CFBI, and unsupervised methods like MATNet and Isomer) provides a temporal modeling foundation for moving instance segmentation, but its performance is also limited by image quality. Event-based motion segmentation research (such as EVIMO and GConv) is hampered by the sparsity of event data, making it difficult to obtain fine instance boundaries. Feature decoupling techniques have been proven effective in separating different semantic factors in fields such as image processing (e.g., image segmentation, super-resolution), but they have not yet been effectively applied in multimodal moving instance segmentation tasks.

[0010] Therefore, existing technologies urgently need an innovative technical solution that can more fully and intelligently mine and fuse complementary information in image and event data to achieve accurate and robust instance segmentation of small-scale moving targets in various extreme and challenging scenarios. Summary of the Invention

[0011] To alleviate or partially alleviate the above-mentioned technical problems, the solution of the present invention is as follows:

[0012] A mobile instance segmentation method, comprising:

[0013] Step S1: Extract image appearance features and image motion features from image frames, and extract event appearance features and event motion features from event stream data;

[0014] Step S2: Perform cross-modal alignment and fusion of the image appearance features and the event appearance features to obtain fused appearance features, and perform cross-modal alignment and fusion of the image motion features and the event motion features to obtain fused motion features;

[0015] The image frames and the event stream data are obtained from the same time window.

[0016] Furthermore, it also includes the following steps:

[0017] Step S3: Interact with the fused appearance features and the fused motion features to generate appearance enhancement features and motion enhancement features;

[0018] Step S4: Input the appearance enhancement features and the motion enhancement features into the task-specific decoder, process the appearance-related tasks and motion-related tasks in parallel, and output the instance segmentation mask and the corresponding motion state label.

[0019] Furthermore, step S1 specifically includes:

[0020] Using a dual-branch image encoder, the image appearance features and the image motion features are extracted from the image frame;

[0021] Using an event dual-branch encoder, the event appearance features and the event motion features are extracted from the preprocessed event voxel representation;

[0022] The image dual-branch encoder includes an image appearance encoder and an image motion encoder; the event dual-branch encoder includes an event appearance encoder and an event motion encoder, wherein the image appearance encoder and the event appearance encoder have the same structure but do not share parameters, and the image motion encoder and the event motion encoder have the same structure but do not share parameters.

[0023] Furthermore, during the training process in step S2, an intramodal contrastive learning loss is introduced to enhance the decoupling between appearance features and motion features within the same modality.

[0024] Furthermore, step S2 specifically includes:

[0025] Distribution alignment step: An adversarial training strategy is adopted, using an appearance domain discriminator to align the distribution of the event appearance features with the distribution of the image appearance features, and using a motion domain discriminator to align the distribution of the image motion features with the distribution of the event motion features.

[0026] Semantic alignment steps: A convolutional mode transformation network is used to train the transformation module to achieve bidirectional transformation between the image appearance features and the event appearance features, as well as bidirectional transformation between the image motion features and the event motion features, and constraints are imposed by reconstruction loss.

[0027] Furthermore, in the distribution alignment step, the feature encoder and the domain discriminator are connected through a gradient inversion layer to achieve simultaneous optimization of feature encoding and domain discrimination during adversarial training.

[0028] Furthermore, the conversion module includes four conversion units, each used for:

[0029] Transform image appearance features into event appearance feature space;

[0030] Transform the event appearance features into the image appearance feature space;

[0031] Transform image motion features into event motion feature space;

[0032] Transform the event motion features into the image motion feature space.

[0033] Furthermore, the task-specific decoder includes:

[0034] An appearance-related decoder is used to process the appearance enhancement features and output an instance segmentation mask and bounding box parameters;

[0035] A motion-related decoder is used to process the motion enhancement features and output motion state labels and dense optical flow fields.

[0036] Furthermore, it also includes the model training step, with the total training loss function being:

[0037] ,in, The loss is used for the main tasks of instance segmentation and motion classification. For intra-modal contrastive learning loss; The overall adversarial loss for cross-modal alignment; Transformation reconstruction loss for cross-modal alignment; For bounding box regression loss; For optical flow reconstruction loss; λ flow , λ bbox , λ con , λ dist and λ sem These are all balance coefficients.

[0038] A mobile instance segmentation system, comprising:

[0039] The data acquisition module is used to acquire synchronized image frames and event stream data;

[0040] The dual decoupled feature extraction module is used to implement step S1: extracting image appearance features and image motion features from image frames, and extracting event appearance features and event motion features from event stream data;

[0041] The cross-modal alignment and fusion module is used to implement step S2: performing cross-modal alignment and fusion of the image appearance features and the event appearance features to obtain fused appearance features, and performing cross-modal alignment and fusion of the image motion features and the event motion features to obtain fused motion features;

[0042] A cross-type interaction module is used to implement step S3: interacting with the fused appearance features and the fused motion features to generate appearance enhancement features and motion enhancement features;

[0043] The task decoding and output module is used to implement step S4: inputting the appearance enhancement features and the motion enhancement features into the task-specific decoder, processing appearance-related tasks and motion-related tasks in parallel, and outputting the instance segmentation mask and the corresponding motion state label.

[0044] The technical solution of this invention has one or more of the following beneficial technical effects:

[0045] (1) Significantly improves the segmentation accuracy of small-scale motion instances. By extracting both appearance and motion features from both images and event streams, the problem of insufficient feature density caused by the sparsity of event data is effectively overcome, and the ability to represent small-sized targets is particularly enhanced.

[0046] (2) Enhance the effectiveness and stability of cross-modal fusion. Achieve dual alignment at the distribution and semantic levels through a multi-granularity cross-modal alignment mechanism, bridge the inherent gap between image and event modalities, ensure the consistency and complementarity of fusion features, and improve the segmentation reliability in complex scenarios.

[0047] (3) It has excellent segmentation performance and generalization ability under extreme conditions. It can significantly improve the instance-level segmentation accuracy for challenging scenarios such as fast movement, low light, and small instance density. The framework is highly compatible with different backbone networks and can achieve excellent performance and efficiency balance through lightweight backbone networks, making it widely applicable.

[0048] Furthermore, other beneficial effects of the present invention will be mentioned in the specific embodiments. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of multimodal feature extraction using existing technologies;

[0050] Figure 2 This is a feature similarity comparison chart between the image modality and the event modality;

[0051] Figure 3 This is a schematic diagram of the multimodal feature extraction of the present invention;

[0052] Figure 4 This is a flowchart illustrating one embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the system architecture of one embodiment of the present invention;

[0054] Figure 6 This is a comparison chart of the qualitative effects of the present invention on the MouseSIS dataset. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order.

[0057] This invention provides a mobile instance segmentation method and system. By innovatively designing a dual decoupling mechanism, it simultaneously extracts both appearance and motion features from images and event streams, effectively solving the problems of sparse event data and insufficient feature representation for small targets. Furthermore, it constructs a multi-granularity cross-modal alignment and fusion framework, combining adversarial training and reversible modality transformation to achieve cross-modal feature distribution and semantic depth alignment, overcoming the feature entanglement and semantic gap problems in traditional fusion methods. Finally, through cross-type interaction and task-specific decoding, it outputs a high-precision instance segmentation mask and motion state, significantly improving the perception robustness and segmentation accuracy of small-scale moving targets in extreme scenarios such as low light and high-speed motion.

[0058] Figure 3 This is a schematic diagram of the multimodal feature extraction of the present invention. As shown in the figure, the present invention differs from the existing paradigm of "image manages appearance, event manages motion" in terms of modal function fixation. Instead, it achieves intramodal feature decoupling by extracting both appearance and motion features from both image and event modalities.

[0059] Figure 4This is a flowchart illustrating one embodiment of the present invention. As shown in the figure, this figure demonstrates the complete flow of the decoupled instance-level moving target segmentation method provided by the present invention. As shown in the figure, the method starts from image frame (I t ) and event flow (E [t, t+Δt] Starting with data input from two modalities, appearance and motion features are extracted within each modality via a dual decoupling layer (containing an image feature decoupling encoder and an event feature decoupling encoder). Subsequently, a cross-modal alignment and fusion layer performs distribution alignment (adversarial training) and semantic alignment (modal reversible transformation) on the extracted corresponding features, and then fuses them. The fused features enter a cross-type interaction layer, which promotes joint reasoning of appearance and motion cues through a cross-attention mechanism. The task decoding layer executes appearance-related and motion-related tasks in parallel, ultimately achieving pixel-level segmentation of all independently moving objects in the scene and predicting the motion state of the objects at the current time step. Specifically, an instance mask m̂ needs to be predicted for each instance. k A binary motion tag ŷ k ∈{0,1}. That is, the final output is the set of instance segmentation results. and motion status label set , where K is the number of instances detected in the current frame.

[0060] Specifically, the input layer contains image frames acquired by a traditional image sensor (I t ), and the asynchronous event stream acquired by the event camera within the corresponding time window [t, t+Δt]. Because the event camera generates an asynchronous event stream E = Each event Encoded at time t i and pixels (x) i , y i The polarity p is at the location. i The brightness variation is represented by N, where N is the number of events. Although event data provides extremely high temporal resolution, it is inherently sparse and irregular, making it unsuitable for dense prediction tasks. To facilitate subsequent processing, the event stream E needs to be discretized into several time bins, and voxel representations are obtained by spatially and temporally accumulating the events.

[0061]

[0062] in, It represents the spatial pixel coordinates at time t and the event voxel value corresponding to the i-th time bin; t is the time step corresponding to the current image frame; t i It represents the specific time when the i-th event occurs; i is the index of the event, corresponding to a single event e. i (Asynchronous event unit output by the event camera); p i The polarity of the i-th event; (xi , y i The spatial pixel coordinates of the i-th event; b∈{0,1,...,B−1} is the temporal bin index, B is the total number of temporal bins; Δt represents the duration of the event segment between the current frame and the next frame. The image modality and the event modality are combined into an input set X= {I t V t}, I t It is the image frame at time t, V t It is the voxel representation of the event stream transformed at time t, and is assigned to the segmentation output through a mapping function f with a learnable parameter ϕ: , The set of instance segmentation results output by the model. This is the set of motion state labels output by the model.

[0063] Figure 5 This is a schematic diagram of the overall architecture of one embodiment of the present invention. As shown in the figure, the architecture of the present invention mainly includes four parts: (1) Dual decoupling mechanism: a dual-branch encoder (appearance encoder and motion encoder) is used to process image and event inputs respectively, and appearance features and motion features are extracted within each modality; (2) Multi-granularity cross-modal alignment and fusion: through a domain discriminator (D a D m Distribution alignment is achieved through a mode converter (T). a1 , T a2 , T m1 , T m2 (2) Semantic alignment is achieved, and the aligned features are fused; (3) Cross-type interaction: joint reasoning between appearance and motion cues is promoted through cross-attention mechanism; (4) Task-specific decoding: decoders for appearance-related tasks (instance segmentation, bounding box regression) and motion-related tasks (motion classification, optical flow estimation) are included for appearance-related and motion-related predictions. The framework outputs the final instance mask and motion state.

[0064] In the dual decoupling mechanism, to ensure sufficient information for small instance segmentation, this invention designs a dual-branch encoder for each modality to extract appearance and motion information from each modality. In the embodiments, the dual-branch encoders for the two modalities share the same input, but their parameters are set independently and trained with different task-specific supervision signals to learn different semantic representations.

[0065] For the image modality, a dual-branch encoder is designed, with one branch serving as the image appearance encoder, specifically used to extract images from the image frame (I...). t Extracting feature representations that are highly correlated with the object's appearance attributes (such as texture, color, and shape) is denoted as... Another branch, as an image motion encoder, aims to infer underlying motion information from cues (such as motion blur) contained in a single frame of an image, and its output features are denoted as... Similarly, for the event modality, a two-branch encoder is also designed. One branch serves as the event appearance encoder, learning from the event voxel V. t The spatial distribution pattern is used to extract the object's outline and surface texture, and the output features are denoted as... Another branch, as an event motion encoder, focuses on extracting clean motion and temporal features from the high temporal resolution characteristics of the event stream, with the output features denoted as... This design allows each modality to contribute both appearance and motion features, greatly enriching the diversity and density of features, and particularly enhancing the representation of small targets. The image and event modalities provide complementary information, alleviating the problem of insufficient dense information within a single modality.

[0066] To further enhance the decoupling between appearance and motion branches, this invention introduces intra-modal contrastive learning. Unlike existing techniques that use contrastive learning to improve cross-modal feature discriminativeness, this invention focuses on the separation of appearance and motion features within a single modality. This prompts the network to emphasize the differences between appearance and motion semantics, rather than differences across modalities, avoiding redundant or mixed representations between branches. For each modality, the positive sample F... ﹢ Selected from the same type (appearance or motion) and consecutive frames, while negative samples F ﹣ Selected from different types or non-continuous frames. The InfoNCE loss for intra-modal contrastive learning is defined as:

[0067]

[0068] Where · represents the dot product between two features normalized by ℓ2 (also known as Euclidean normalization), and τ is the temperature factor. The specific construction details of positive and negative samples of appearance and motion features are existing technologies and will not be elaborated here.

[0069] Since image and event data originate from different imaging mechanisms, their features differ in statistical distribution and semantic space, resulting in limited direct fusion effects. To ensure semantic consistency and fully utilize the complementarity between image and event data, this invention designs a multi-granularity cross-modal alignment module that includes distribution alignment and semantic alignment, as well as a lightweight convolutional neural network (CNN) fusion layer for image and event modalities. The four sets of decoupled features obtained through the dual decoupling mechanism ( , , , By strengthening the consistency between the distribution level and the semantic level during the feature fusion process, the appearance and motion features of different modalities can be aligned.

[0070] First, a distribution alignment step is performed. Since images provide more explicit appearance cues, while events contain clearer motion cues, this invention employs an asymmetric strategy for distribution alignment. Image features serve as the reference domain for appearance alignment, while event features serve as the reference domain for motion alignment. Two discriminators are introduced for both the appearance and motion branches. The discriminators classify the feature modalities, while the encoder learns to narrow this gap through a gradient inversion layer. Specifically, for appearance features, image appearance features are used as the reference domain for motion alignment. Using the distribution of as the target domain (reference distribution), an appearance domain discriminator D is introduced. a During the training process, D a Attempting to distinguish whether the input features come from still The appearance encoders for images and events are jointly trained via a gradient reversal layer (GRL) to generate a model capable of "fooling" the appearance domain discriminator D. a The characteristic that forces Distribution direction Proximity. Similarly, for motion features, use event motion features. The distribution of is used as the target domain, and a motion domain discriminator D is introduced. m and prompt Distribution direction Alignment, overall adversarial loss is , To combat losses in appearance branches, To counteract the loss of the movement branch.

[0071] Each component corresponds to an adversarial training objective between the encoder and its corresponding domain discriminator. The overall adversarial loss can be expressed as a min-max optimization as follows:

[0072]

[0073] Where G represents the feature encoder, D represents the domain discriminator, and p ref p represents the reference domain distribution (the reference domain is the domain-aligned "target distribution"). src Let x represent the source domain distribution (the source domain is the feature distribution that needs to be aligned), and let z represent samples that follow the reference domain distribution Pref. src The sample. For the appearance branch, and For the branches of motion, and .

[0074] Next, a semantic alignment step is performed. To ensure the consistency of similar features across different modalities at the high-level semantic level, this invention introduces a lightweight convolutional modality transfer network (CMT network) to transform appearance and motion features between image space and event space. The transformation module (CMT network) contains four transformation units: T a1 Learning will Transform to the event appearance feature space, T a2 Learning will Transform to image appearance feature space; T m1 and T m2 These are the conversion modules for motion features. T m1 Will Transform to the event motion feature space, T m2 Will Transform to the image motion feature space. During training, use reconstruction loss. (like Figure 5 The features transformed by the L2 loss constraint should be able to accurately reconstruct the corresponding cross-modal features. For example, it is required that T a1 ( (close to) T a2 ( (close to) The motion characteristics are similar. In other words, T a1 T a2 T m1 and T m2 All are modality transformation units, capable of performing bidirectional feature reconstruction to achieve semantic alignment. The reconstruction loss is shown below, and this process is regularized:

[0075]

[0076] Where T represents the transformation module. This design ensures that features of the same semantic type can be transformed into each other, strengthens cross-modal alignment, and provides a more stable foundation for fusion. Importantly, both distribution alignment and semantic alignment are completely unsupervised, applied only during the training phase, and do not add any additional overhead to inference time.

[0077] After alignment, the appearance features from the two modalities are fused (e.g., by concatenating channels and then adding a lightweight convolutional layer) to obtain fused appearance features; the motion features from the two modalities are fused to obtain fused motion features.

[0078] The fused features enter a cross-type interaction layer. This layer aims to promote synergy and complementarity between appearance and motion information. This embodiment employs a cross-attention mechanism to achieve this interaction. The two enhanced features can then be further fused or sent separately to subsequent decoders.

[0079] Finally, the processed features are fed into the task decoding layer. To preserve the semantics of the decoupled features, this invention designs instance segmentation and bounding box regression for the decoder of appearance-related tasks, and motion classification and optical flow estimation for the decoder of motion-related tasks. This layer contains appearance-related decoders and motion-related decoders, which handle different sub-tasks respectively. The appearance-related decoder is responsible for generating candidate bounding box parameters b for each instance. k And precise pixel-level binary segmentation mask m̂ k The motion-related decoder is used to generate binary motion tags to determine the motion state of each instance. k The system uses an optical flow decoding network ∈{0,1} and an optional optical flow decoding network to estimate the dense optical flow field of the scene. This optical flow estimation can be used as an additional self-supervised signal to assist network training. The final output is a set of instance segmentation results. and motion status label set , where K is the number of instances detected in the current frame.

[0080] The overall training objective of this invention integrates task-specific supervision, intra-modal contrastive learning, and cross-modal alignment. Total loss function. It is a weighted sum of multiple monitoring signals.

[0081] The main task loss for instance segmentation and motion classification:

[0082]

[0083] Among them, y k and ŷ k These are the actual motion labels and the predicted motion labels, m k and m̂ k These represent the true mask and the predicted mask, respectively. It is the standard cross-entropy loss used for class prediction. It is the binary cross-entropy loss used for mask supervision, where K is the number of instances detected in the current frame.

[0084] Additional task-specific losses include motion task loss (optical flow reconstruction loss):

[0085]

[0086] in, This represents the optical flow between two adjacent frames predicted by a stream decoder based on a Feature Pyramid Network (FPN). and This represents the pixel intensity of two consecutive frames, treated as a continuous function of spatial coordinates c. (Item) The next frame is sampled at the displacement position determined by the estimated flow, effectively warping it toward the current frame. The robust function ψ(·) follows... This includes ϵ = 0.01 and q = 0.4, where u is the input value of the function (i.e., the pixel intensity difference within the parentheses).

[0087] Additional task-specific loss includes appearance task loss (bounding box regression loss): , where c b Indicates the reference coordinates of the bounding box. This represents the predicted bounding box coordinates.

[0088] The contrastive loss mentioned above is used to enhance intra-modal feature decoupling. Achieving overall adversarial loss for cross-modal alignment and transformation reconstruction loss And appearance task loss (bounding box regression loss) ) and motion task loss (optical flow reconstruction loss) These are combined. By adjusting the weight coefficients λ of each item, different optimization objectives are balanced. That is:

[0089]

[0090] Where, λ flow , λ bbox , λ con , λ dist and λ sem It is the balance coefficient.

[0091] This invention, through the above-described specific embodiments, fully discloses an innovative mobile instance segmentation technology solution. This solution systematically constructs a processing flow of "intramodal decoupling - cross-modal alignment - cross-type interaction," effectively overcoming the shortcomings of existing technologies such as sparse features, entanglement, and inefficient fusion. It provides a complete and feasible technical path for achieving robust and accurate instance-level motion perception in extreme environments. Those skilled in the art will understand that the specific implementation methods of the encoder skeleton, fusion module, and combination of loss functions in the above embodiments can be selected and adjusted according to actual needs without departing from the basic principles and core concepts of this invention.

[0092] To verify the effectiveness of this invention, extensive experiments were conducted on three benchmark datasets: MouseSIS, SEVD-Fixed, and EVIMO. MouseSIS contains synchronized grayscale frames and an event stream of interactive mice, along with over 75,000 temporally consistent instance masks. The target objects are small and frequently occluded, making it suitable for evaluating fine-grained segmentation of objects with low foreground proportions. SEVD-Fixed is a synthetic traffic monitoring dataset containing RGB, event, depth, and semantic labels captured under different lighting and weather conditions. Foreground objects such as vehicles and pedestrians are typically small, posing a challenge to accurate instance segmentation. EVIMO provides an indoor event stream with realistic motion masks and depth, containing up to three moving objects, serving as a standard benchmark for motion segmentation.

[0093] Table 1: Summary of the three datasets

[0094]

[0095] As shown in Table 1, compared with EVIMO (3.74%), MouseSIS and SEVD-Fixed have lower average instance mask area ratios (0.73% and 0.15%, respectively) and lower foreground coverage. This highlights the particular challenges they face in small instance segmentation.

[0096] In the specific implementation, the framework of this invention is implemented in PyTorch. For the aforementioned three datasets, the network is trained for 400K iterations on MouseSIS, 500K iterations on EVIMO, and 800K iterations on SEVD-Fixed, with a batch size of 16. The Adam optimizer is used, with a weight decay of 1×10⁻⁶. -6 It employs single-cycle learning rate scheduling, with a peak learning rate set to 1×10⁻⁶. -4 In all experiments, the number of event boxes was set to B=10, and the moving confidence threshold was set to θ=0.1. The loss weight was set to λ. flow = 10.0, λ con = 0.5, λ bbox = 0.01, λ dist = 0.1, λ sem = 10.0. All experiments, including ablation experiments and comparisons with previous methods, were performed on the same evaluation machine. Training was performed on dual A40 GPUs, and inference was performed on a single RTX 5090 GPU to ensure consistent evaluation settings.

[0097] Because most existing event-based sensors (event cameras) have relatively low spatial resolution, and consistent resolution is required across different modalities, different datasets are downsampled to a specific target resolution during implementation. The input resolution for MouseSIS is adjusted to 320×180, and for SEVD-Fixed to 512×384. This downsampling strategy significantly reduces computational overhead while preserving instance-level discriminability (especially for small targets). For EVIMO, the original resolution of 346×260 is used. Importantly, the lower resolution setting further highlights the challenges of segmenting small instances.

[0098] Generally, three main metrics are used to evaluate motion instance segmentation: mIoU ins , mIoU 01 And mAP. Specifically, mIoU ins Evaluate the instance-level segmentation accuracy for each moving object, while mIoU 01 Measure the accuracy of the 0-1 binary foreground mask. Additionally, report mAP to account for false positives and overall detection accuracy.

[0099] The framework of this invention was quantitatively compared with representative frame-based methods (IDOL) and event-assisted methods (ModelMixSort and EvInsMos) on three challenging benchmark datasets (MouseSIS, SEVD-Fixed, and EVIMO). The results are summarized in Table 2.

[0100] Table 2: Quantitative comparisons on MouseSIS, SEVD-Fixed, and EVIMO

[0101]

[0102] On the MouseSIS dataset, the proposed method achieves the best instance-level segmentation accuracy of 70.25%, outperforming traditional frame-based methods (IDOL) and event-assisted baseline methods. Notably, ModelMixSort EvInsMOS has improved upon pure image methods, demonstrating the advantage of utilizing event data under challenging lighting conditions. The proposed method further enhances performance through explicit decoupling and alignment strategies. On the SEVD-Fixed dataset (more challenging than MouseSIS due to its complex outdoor environment, diverse weather conditions, and more small instances), the proposed framework achieves 62.05%, 5.55% higher than EvInsMOS. This sustained performance improvement highlights the superior robustness of the proposed method under extreme conditions, where event signals effectively complement degraded image data. On the EVIMO dataset, although EvInsMOS and ModelMixSort have already demonstrated strong performance by utilizing two modalities, the proposed method still achieves the highest accuracy (72.08%), indicating the effectiveness of the proposed decoupling strategy.

[0103] Overall, these results demonstrate that the proposed method achieves continuous improvement on benchmark datasets with varying lighting conditions, motion patterns, and scene complexities. Importantly, both MouseSIS and SEVD-Fixed contain a large number of small instances, whose accurate segmentation heavily relies on dense appearance and motion features. The decoupling framework of this invention effectively enhances the segmentation of such small instances by simultaneously extracting appearance and motion information from both modalities. In contrast, conventional frame-based methods or simple fusion methods do not explicitly perform this bimodal decoupling, highlighting the advantage of the proposed method in jointly utilizing complementary appearance and motion cues.

[0104] Figure 6 This is a comparison of the qualitative performance of this invention on the MouseSIS dataset. As shown in the figure, the visual comparison displays consecutive frames sampled from the MouseSIS video sequence, arranged from top to bottom. Rectangular boxes highlight areas where segmentation was missed or where boundaries were unclear. The figure includes segmentation results for images, events, ground truth (GT), IDOL, ModelMixSort, EvInsMOS, and this invention (DIMOS). The results consistently show that the DIMOS framework achieves more accurate and temporally consistent segmentation, especially for small motion instances.

[0105] Compared to IDOL, this invention produces clearer target boundaries and avoids missed detections under motion blur or low-light conditions. The superiority of this method is particularly evident in scenarios where pure image models often struggle to distinguish target contours or confuse overlapping instances. Event-assisted baseline methods, including ModelMixSort and EvInsMOS, perform better by utilizing event information; however, they still suffer from mask fragmentation or inaccurate segmentation when multiple targets move close together or interact. Overall, visual comparisons and quantitative results are in high agreement, confirming the superior robustness of this invention's framework for segmenting small instances at finite resolution.

[0106] To investigate the contribution of each component, ablation experiments were conducted on MouseSIS, progressively enabling task-specific supervision, dual decoupling mechanisms, alignment modules, and the encoder backbone network. The results are shown in Tables 3 and 4.

[0107] Table 3: Ablation experiments on MouseSIS

[0108]

[0109] In this invention, a dual decoupling mechanism (Dual. Mech.), additional task-specific losses (UnFlow and BBox), semantic alignment loss (Sem. Align.), and distribution alignment loss (Dist. Align.) are gradually added.

[0110] Table 4: Backbone network ablation experiments on the MouseSIS dataset

[0111]

[0112] As shown in Table 3, without any additional modules, the model simplifies to a simple multimodal interaction pipeline with a success rate of only 60.47%, which confirms that insufficient feature extraction makes it difficult to handle small instance segmentation under challenging conditions.

[0113] As shown in Table 3, introducing unsupervised flow estimation improves the accuracy from 60.47% to 62.54%, indicating that additional motion guidance helps capture motion cues. Combining bounding box supervision further improves it to 63.46%, providing spatial priors and enhancing localization capabilities (especially for small or overlapping targets). For sequences without bounding box annotations, pseudo-boundary boxes are generated from the outer boundaries of instance masks.

[0114] Table 3 also shows that the dual decoupling mechanism significantly improves performance by 68.11%. This module explicitly separates appearance and motion information in the two modalities, resulting in denser and more discriminative features after cross-modal fusion, which is particularly beneficial for small instance segmentation. By introducing intra-modal contrastive learning for explicit decoupling, the network avoids mutual interference between appearance and motion semantics and ensures that each branch focuses on its corresponding representation.

[0115] Adding semantic alignment further improves performance to 69.23%, indicating that cross-modal transformation of corresponding appearance and motion features enhances the fusion effect. Furthermore, enabling distribution alignment improves performance to 70.25%. This is because features extracted from images and event streams naturally follow different distributions, which could potentially degrade performance. Semantic reconstruction and adversarial domain adaptation together ensure consistency in representations across different modalities.

[0116] Table 4 shows the results using different backbone networks on MouseSIS, confirming that the dual decoupling and alignment module of this invention can effectively utilize the representational capabilities of deeper networks. Importantly, using lightweight backbone networks (such as MobileNetV2 and ResNet-18) only results in a small performance drop of 1.63% and 0.93%, respectively, indicating that the framework of this invention has strong backbone-independent generalization capabilities. This suggests that the performance improvement comes from this module, rather than solely relying on a large encoder. This characteristic is particularly advantageous considering that the decoupling framework of this invention involves multiple encoder branches. By employing dual lightweight backbone networks, the overall parameters can be reduced while maintaining or even surpassing the performance of traditional methods that rely on a single large-capacity backbone network (e.g., approximately 7.0M parameters for dual MobileNetV2, and approximately 25.6M parameters for a single ResNet-50), achieving a good performance-efficiency tradeoff.

[0117] This invention addresses the challenge of segmenting small moving instances. Within each modality, appearance and motion representations are decoupled and extracted, and then aligned using a multi-granularity cross-modal alignment strategy. This design improves feature density and fusion performance, achieving continuous improvement across multiple datasets. Experimental results demonstrate the effectiveness and robustness of the proposed method, particularly in segmenting small instances under challenging conditions.

[0118] To better illustrate the present invention, numerous specific details have been provided in the detailed embodiments described above. Those skilled in the art should understand that the present invention can be practiced even without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of the present invention.

[0119] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for segmenting mobile instances, characterized in that, include: Step S1: Extract image appearance features and image motion features from image frames, and extract event appearance features and event motion features from event stream data; Step S2: Perform cross-modal alignment and fusion of the image appearance features and the event appearance features to obtain fused appearance features, and perform cross-modal alignment and fusion of the image motion features and the event motion features to obtain fused motion features; The image frames and the event stream data are obtained from the same time window; Step S1 specifically includes: using an image dual-branch encoder to extract the image appearance features and the image motion features from the image frame; using an event dual-branch encoder to extract the event appearance features and the event motion features from the preprocessed event voxel representation; wherein, the image dual-branch encoder includes an image appearance encoder and an image motion encoder; the event dual-branch encoder includes an event appearance encoder and an event motion encoder, and the image appearance encoder and the event appearance encoder have the same structure but do not share parameters, and the image motion encoder and the event motion encoder have the same structure but do not share parameters; Step S3: Interact with the fused appearance features and the fused motion features to generate appearance enhancement features and motion enhancement features; Step S4: Input the appearance enhancement features and the motion enhancement features into the task-specific decoder, process the appearance-related tasks and motion-related tasks in parallel, and output the instance segmentation mask and the corresponding motion state label.

2. The mobile instance segmentation method according to claim 1, characterized in that: During the training process in step S2, an intramodal contrastive learning loss is introduced to enhance the decoupling between appearance features and motion features within the same modality.

3. The mobile instance segmentation method according to claim 1, characterized in that, Step S2 specifically includes: Distribution alignment step: An adversarial training strategy is adopted, using an appearance domain discriminator to align the distribution of the event appearance features with the distribution of the image appearance features, and using a motion domain discriminator to align the distribution of the image motion features with the distribution of the event motion features. Semantic alignment steps: A convolutional mode transformation network is used to train the transformation module to achieve bidirectional transformation between the image appearance features and the event appearance features, as well as bidirectional transformation between the image motion features and the event motion features, and constraints are imposed by reconstruction loss.

4. The mobile instance segmentation method according to claim 3, characterized in that: In the distribution alignment step, the feature encoder and the domain discriminator are connected through a gradient inversion layer to achieve simultaneous optimization of feature encoding and domain discrimination during adversarial training.

5. The mobile instance segmentation method according to claim 3, characterized in that, The conversion module includes four conversion units, which are used for: Transform image appearance features into event appearance feature space; Transform the event appearance features into the image appearance feature space; Transform image motion features into event motion feature space; Transform the event motion features into the image motion feature space.

6. The mobile instance segmentation method according to claim 1, characterized in that, The task-specific decoder includes: An appearance-related decoder is used to process the appearance enhancement features and output an instance segmentation mask and bounding box parameters; A motion-related decoder is used to process the motion enhancement features and output motion state labels and dense optical flow fields.

7. The mobile instance segmentation method according to any one of claims 1-6, characterized in that, It also includes the model training step, and the total training loss function is: ,in, The loss is used for the main tasks of instance segmentation and motion classification. For intra-modal contrastive learning loss; The overall adversarial loss for cross-modal alignment; Transformation reconstruction loss for cross-modal alignment; For bounding box regression loss; For optical flow reconstruction loss; λ flow , λ bbox , λ con , λ dist and λ sem These are all balance coefficients.

8. A mobile instance segmentation system, characterized in that, include: The data acquisition module is used to acquire synchronized image frames and event stream data; The dual decoupled feature extraction module is used to implement step S1: extracting image appearance features and image motion features from image frames, and extracting event appearance features and event motion features from event stream data; Step S1 specifically includes: using an image dual-branch encoder to extract the image appearance features and the image motion features from the image frame; using an event dual-branch encoder to extract the event appearance features and the event motion features from the preprocessed event voxel representation; wherein, the image dual-branch encoder includes an image appearance encoder and an image motion encoder; the event dual-branch encoder includes an event appearance encoder and an event motion encoder, and the image appearance encoder and the event appearance encoder have the same structure but do not share parameters, and the image motion encoder and the event motion encoder have the same structure but do not share parameters; The cross-modal alignment and fusion module is used to implement step S2: performing cross-modal alignment and fusion of the image appearance features and the event appearance features to obtain fused appearance features, and performing cross-modal alignment and fusion of the image motion features and the event motion features to obtain fused motion features; A cross-type interaction module is used to implement step S3: interacting with the fused appearance features and the fused motion features to generate appearance enhancement features and motion enhancement features; The task decoding and output module is used to implement step S4: inputting the appearance enhancement features and the motion enhancement features into the task-specific decoder, processing appearance-related tasks and motion-related tasks in parallel, and outputting the instance segmentation mask and the corresponding motion state label.

Images