A scene segmentation method and system based on multi-modal small sample learning

Abstract

The application discloses a scene segmentation method and system based on multi-modal small sample learning, and relates to the technical field of computer vision. The method comprises the following steps: acquiring an automatic driving video stream, and extracting image frames in the video stream; training and testing a scene segmentation model by using a data set to obtain a trained scene segmentation model, wherein a lightweight backbone network is used to extract semantic features of the data set, then the scene segmentation model is used to extract double-axis self-attention features and correct information of the image frames after the semantic feature extraction, and finally, the corrected multi-source features are fused to obtain a prediction output graph; and the trained scene segmentation model is used for scene segmentation processing on a real-time video stream to be segmented. The application can effectively enhance the understanding ability of semantic information in a complex scene while ensuring the lightweight of the model, and significantly improve the segmentation precision and generalization ability under the condition of small samples.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a scene segmentation method and system based on multimodal few-shot learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence and deep learning technologies, scene segmentation, as a key task in image semantic understanding, has been widely applied in various fields such as autonomous driving, intelligent security, human-computer interaction, assisted diagnosis, and robot perception. Its goal is to accurately classify each pixel in an image into its corresponding semantic category, thereby achieving a structured understanding of complex environments.

[0004] Traditional scene segmentation methods mostly rely on single-modal visual information, typically employing architectures such as convolutional neural networks (CNNs) or visual transformers to perform pixel-level classification of RGB images. While these methods have achieved good results on standard datasets, they face numerous challenges in practical applications. For example, factors such as changes in lighting, occlusion, image blurring, and similarity in visual features between categories often reduce the robustness of single-modal methods, limiting their generalization capabilities.

[0005] To improve the generalization ability and semantic understanding of segmentation models in real-world environments, researchers have gradually introduced multimodal information fusion strategies, especially combining textual information such as semantic labels, image descriptions, and task instructions with visual information. This allows the model to not only "see" but also "understand." Through the contextual information provided by natural language, the model can more accurately understand the semantic boundaries of targets in an image, improving classification accuracy. For example, when identifying "a person walking" and "a person standing" in an image, the language description can provide cues about actions or states, thereby assisting the model in achieving more refined semantic segmentation.

[0006] However, the implementation of multimodal methods typically relies on large-scale labeled samples for model training, such as image-text pairs and image segmentation masks. This high data dependence presents significant bottlenecks in practical applications: acquiring large-scale, high-quality multimodal datasets is expensive, and the annotation process is complex and time-consuming, especially in fine-grained tasks or specific fields such as autonomous driving, medical imaging, and remote sensing images. In autonomous driving scenarios, high-precision image segmentation and semantic understanding are particularly needed, requiring the acquisition of a large number of finely labeled images and corresponding environmental information. However, the annotation cost for autonomous driving data is extremely high, and environmental changes, such as weather, lighting, and traffic density, lead to diverse data distributions, making the collection and processing of large-scale labeled data even more challenging. Therefore, how to efficiently train under small sample conditions has become an important research direction.

[0007] In autonomous driving, real-time performance is crucial, requiring the system to quickly analyze and respond to road conditions. Therefore, lightweight and efficient models are needed. However, current mainstream multimodal scene segmentation networks generally have complex structures, large numbers of model parameters, and high computational resource consumption during training and inference, making them difficult to meet the high real-time and resource-friendly requirements of application scenarios such as edge computing and mobile deployment.

[0008] In summary, how to overcome the shortcomings of existing multimodal scene segmentation methods, such as unsatisfactory training results with small samples, poor real-time performance, and low resource utilization, and enable them to be efficiently applied in complex environments such as autonomous driving, has become an urgent problem to be solved by existing technologies. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the present invention aims to provide a scene segmentation method and system based on multimodal few-shot learning, which has advantages such as lightweight, high accuracy, and strong generalization ability, and is suitable for semantic understanding tasks in resource-sensitive terminal devices and complex environments.

[0010] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0011] The first aspect of this invention provides a scene segmentation method based on multimodal few-shot learning, comprising the following steps:

[0012] Acquire the autonomous driving video stream, extract image frames from the video stream, and perform data augmentation on the image frames to obtain the dataset;

[0013] The scene segmentation model was trained and tested using the dataset to obtain the trained scene segmentation model. The model used a lightweight backbone network to extract semantic features from the dataset. Then, the scene segmentation model was used to perform dual-axis self-attention feature extraction and information correction on the image frames after semantic feature extraction. Finally, the corrected multi-source features were fused to obtain the predicted output image.

[0014] The trained scene segmentation model is used to perform scene segmentation processing on the real-time video stream to be segmented.

[0015] Furthermore, the specific steps for extracting image frames from the video stream and performing data augmentation on the image frames are as follows:

[0016] Image frames are extracted from video streams of different driving scenarios at a fixed frame rate, and the image frames are labeled.

[0017] Perform multi-strategy joint data augmentation processing on image frames;

[0018] The enhanced image frames are normalized to obtain the dataset;

[0019] The dataset is divided into a training set and a test set.

[0020] Furthermore, the lightweight backbone network parameters have been pre-frozen.

[0021] Furthermore, the specific steps for performing dual-axis self-attention feature extraction on the image frames after semantic feature extraction using the scene segmentation model are as follows:

[0022] The image frame is enhanced from multiple aspects using three feature branches;

[0023] The feature maps obtained from the three feature branches are fused to obtain the fused feature.

[0024] Furthermore, the three feature branches are the directional branch, the original branch, and the global branch. The directional branch is used to extract directional weights, the original branch is used to preserve basic texture information, and the global branch expands the receptive field through depthwise separable convolution.

[0025] Furthermore, the specific steps for information correction processing of image frames after semantic feature extraction using a scene segmentation model are as follows:

[0026] The fused features are fed into the channel correction unit and the spatial correction unit for correction, resulting in channel correction features and spatial correction features, respectively.

[0027] The final calibration feature is obtained by combining the channel correction feature and the spatial correction feature.

[0028] Furthermore, the specific steps for fusing the corrected multi-source features are as follows:

[0029] A spatial attention mechanism is applied to the calibration features to enhance the response of salient regions in the spatial dimension and highlight the location information of key targets;

[0030] A cross-attention mechanism is introduced to interactively model spatially enhanced image features with auxiliary features from other modalities or levels, thereby achieving the fusion and reconstruction of multi-source semantic information through mutual guidance between features;

[0031] The final output features are decoded to obtain the prediction mask.

[0032] A second aspect of the present invention provides a scene segmentation system based on multimodal few-shot learning, comprising:

[0033] The data acquisition module is configured to acquire autonomous driving video streams, extract image frames from the video streams, and perform data augmentation processing on the image frames to obtain a dataset.

[0034] The model training module is configured to train and test the scene segmentation model using the dataset to obtain the trained scene segmentation model. The module uses a lightweight backbone network to extract semantic features from the dataset, then uses the scene segmentation model to perform dual-axis self-attention feature extraction and information correction on the image frames after semantic feature extraction, and finally fuses the corrected multi-source features to obtain the predicted output image.

[0035] The scene segmentation module is configured to use the trained scene segmentation model to perform scene segmentation processing on the real-time video stream to be segmented.

[0036] A third aspect of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing steps in the scene segmentation method based on multimodal few-shot learning as described in the first aspect of the present invention.

[0037] A fourth aspect of the present invention provides a computer device comprising:

[0038] A processor, adapted to execute computer programs;

[0039] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the scene segmentation method based on multimodal few-shot learning as described in the first aspect of the present invention.

[0040] The above one or more technical solutions have the following beneficial effects:

[0041] This invention discloses a scene segmentation method and system based on multimodal few-shot learning. To address the problems of low accuracy, large and difficult-to-deploy models, and insufficient multimodal information fusion in existing scene segmentation methods under few-shot conditions, this invention proposes a few-shot scene segmentation method that integrates visual and textual information. This method boasts advantages such as lightweight design, high accuracy, and strong generalization ability, making it suitable for semantic understanding tasks in resource-sensitive terminal devices and complex environments. This invention extracts keyframes from in-vehicle videos and constructs image-text pairs by combining sensor data and natural language descriptions, achieving joint modeling of image and semantic information. In the data preprocessing stage, multi-strategy data augmentation and pixel normalization methods are employed to effectively expand sample diversity and alleviate overfitting problems under few-shot conditions. In terms of model design, the first step involves extracting image features using an encoder and feeding them into a dual-axis self-attention feature module (BAFM). This module models contextual information at the horizontal, vertical, and global levels, fusing them to form more discriminative deep features. Next, an information correction module (ICM) guides the enhanced features to semantically align with textual information in both spatial and channel dimensions, improving the consistency and accuracy of cross-modal fusion. Subsequently, the features are fed into a multi-source feature fusion module (MFFM), which introduces spatial attention and cross-attention mechanisms to achieve full fusion of features at different scales and modalities, enhancing the perception of fine-grained targets. Finally, a decoder combining a Transformer structure is used to model global dependencies, and pixel-level branches are used to restore image boundaries and details. The output segmented image is then restored to its original size using bilinear interpolation, ensuring clear edges and semantic consistency. This invention effectively improves segmentation accuracy and generalization ability in complex scenes while maintaining a lightweight model.

[0042] This invention addresses the problem of limited training sample size by constructing a data preprocessing mechanism to enhance perception capabilities. This mechanism expands the dataset, increases data diversity, and prevents overfitting due to small sample sizes.

[0043] This invention proposes a segmentation network composed of a vision-language multimodal collaborative modeling mechanism. The network first extracts multi-scale features from both the support and query images using a backbone network, and then enhances salient regions using support masks. Next, image features are input to mine region correlations. Textual information is mapped to low-dimensional semantic vectors through Word2Vec vector embedding, achieving feature alignment between language and visual modalities and improving semantic consistency. Finally, the system integrates image structural features and linguistic semantic information at the spatial level, outputting a final predicted image with semantic information, thus completing the entire segmentation process.

[0044] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of a scene segmentation method based on multimodal few-shot learning in an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the image preprocessing method in Embodiment 1 of the present invention;

[0048] Figure 3 This is a schematic diagram of the network framework based on multimodal few-shot learning in Embodiment 1 of the present invention;

[0049] Figure 4 This is a schematic diagram of BAFM in Embodiment 1 of the present invention;

[0050] Figure 5 This is a schematic diagram of the ICM in Embodiment 1 of the present invention;

[0051] Figure 6 This is a schematic diagram of MFFM in Embodiment 1 of the present invention. Detailed Implementation

[0052] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0053] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0055] Example 1:

[0056] Embodiment 1 of the present invention provides a scene segmentation method based on multimodal few-shot learning, such as... Figure 1 As shown, it includes the following steps:

[0057] Step 1: Obtain the autonomous driving video stream, extract the image frames from the video stream, and perform data augmentation processing on the image frames to obtain the dataset.

[0058] Step 1.1: Acquire autonomous driving video streams from autonomous driving scenarios.

[0059] Step 1.2: Extract image frames from the video stream and perform data augmentation processing on the image frames.

[0060] Step 1.2.1: Extract image frames from video streams of different driving scenarios at a fixed frame rate and label the image frames.

[0061] In one specific implementation, an autonomous driving video stream is first received, and key image frames are extracted from different driving scenarios at a fixed frame rate. Simultaneously, image-text pairs are automatically constructed by combining vehicle sensor data and natural language descriptions to enhance multimodal information for scene understanding. Specifically, frame-by-frame semantic annotation is performed using manual or assisted annotation tools, assigning clear semantic category labels to each image frame to build a high-quality training and testing dataset. Since the video consists of consecutive image frames, an image can be extracted at a fixed interval (e.g., every 10 frames), thus converting the video sequence into static images for use as input to the model.

[0062] In the image annotation stage, each pixel location is assigned a corresponding category label and represented by a specific numerical encoding method, ultimately generating a binary mask map that corresponds one-to-one with the image, which is used to guide the model to learn the semantic structure in the image.

[0063] Step 1.2.2: Perform multi-strategy joint data augmentation processing on the image frames.

[0064] In one specific implementation, multi-strategy joint data augmentation operations are performed on image data in autonomous driving scenarios, including but not limited to basic augmentation methods such as random cropping, mirror flipping, color perturbation, blurring, and noise superposition, in order to improve the robustness of the model under various environmental changes (such as different weather and lighting conditions) and alleviate the overfitting problem caused by small samples.

[0065] Specifically, such as Figure 2As shown, to improve the model's generalization ability under limited sample conditions, various enhancement strategies are introduced into the constructed dataset to process the input images and their labeled images, thereby expanding the data scale and effectively reducing the model's false detection rate. Considering the manual cost and labeling error caused by pixel-by-pixel annotation, combining image enhancement and preprocessing techniques can significantly alleviate this problem, improving network performance while reducing dependence on massive amounts of data.

[0066] First, the image frames are blurred. Specifically, a Gaussian filter is applied to perform a convolution operation on the image matrix to smooth image edges and reduce the intensity of detail textures; the corresponding annotation map remains unchanged to ensure semantic alignment.

[0067] Subsequently, artificial noise is superimposed on the image. Specifically, Gaussian noise or salt-and-pepper noise is added to some pixels with a set probability to simulate perceptual anomalies or sensor interference by randomly perturbing the image pixel values.

[0068] The two enhancement methods mentioned above are automatically combined and invoked in each training iteration to achieve cross-enhancement of the data. The size of the blur kernel and the range of noise intensity are determined according to the preset strategy, and different versions of perturbation images are randomly generated each time images are loaded.

[0069] Step 1.2.3: Normalize the enhanced image frames.

[0070] In one specific implementation, the image pixels are normalized to map the pixel values ​​to a standardized range, eliminating the distribution drift caused by the enhancement operation, ensuring that the image input is adapted to the subsequent neural network, thereby improving the accuracy and stability of the model in complex driving scenarios.

[0071] Step 1.2.4: Divide the dataset into training and test sets.

[0072] Step 2: Use the dataset to train and test the scene segmentation model to obtain the trained scene segmentation model.

[0073] The process involves using a lightweight backbone network to extract semantic features from the dataset, followed by a scene segmentation model to perform dual-axis self-attention feature extraction and information correction on the extracted image frames. Finally, the corrected multi-source features are fused to obtain the predicted output image. The specific steps are as follows:

[0074] Step 2.1: Extract semantic features from the dataset using a lightweight backbone network. To avoid overfitting during training, the lightweight backbone network parameters are pre-frozen and no gradient updates are performed to stabilize the overall feature representation structure.

[0075] In one specific implementation, the scene segmentation model uses ResNet50 as the backbone network. Features of the input image are extracted through the backbone network, and the model is built with the parameters of the backbone model frozen. The enhanced image is input to the lightweight backbone network to extract multi-scale semantic features. A parameter freezing strategy is used to maintain the stability of low-level features, and the model is divided into support set and query set for training.

[0076] In autonomous driving scene segmentation tasks, semantic features typically refer to the semantic category information of each pixel in an image, such as the feature information embodied by elements like roads, vehicles, pedestrians, buildings, traffic signs, and the sky. These features are automatically extracted from images through deep neural networks, containing both spatial location information and high-level semantic understanding capabilities. This helps the model accurately identify and distinguish different types of targets, achieving comprehensive perception and understanding of complex traffic environments. This embodiment achieves multi-scale semantic feature extraction by setting a backbone network with frozen parameters. By preserving details at the shallow level and extracting semantics at the deep level, a multi-scale feature representation is formed, providing a strong data foundation for subsequent data processing in the scene segmentation model.

[0077] After preprocessing the dataset by extracting semantic features using a lightweight backbone network, the training set of road scene images is divided into a support set and a query set and input into the scene segmentation model for training. To eliminate irrelevant background interference, a Transformer structure and convolution operators are introduced to fuse image features and enhance the complementary expression of local and global semantic information. This ensures both speed improvement and effective accuracy enhancement.

[0078] Specifically, the preprocessed road scene image training set is input... Figure 3The network structure shown is used for training. The scene segmentation model in this embodiment is an architecture combining a Transformer structure and convolutional operators. During training, the encoder first extracts image semantic features and feeds them into a Bi-Axial Feature Attention Module (BAFM) to form context-rich feature representations. Subsequently, an Information Calibration Module (ICM) performs spatial and semantic alignment on the fused features to improve feature consistency. Based on this, the features are fed into a Multi-Source Feature Fusion Module (MFFM), which fuses co-occurring information and similarity vectors through a multi-branch mechanism, effectively alleviating the problem of uneven distribution of multimodal information and further integrating features at different scales to improve the perception of small targets. Finally, a Transformer decoder models global dependencies, while a pixel-level decoder restores image edges and details, jointly outputting the final segmentation prediction result. This design effectively improves the accuracy and boundary clarity of semantic segmentation in autonomous driving scenarios while ensuring real-time performance.

[0079] In terms of network structure, this embodiment innovatively introduces a skip connection mechanism to achieve multi-layer semantic and spatial information fusion between the encoder and decoder, effectively improving the model's accuracy and inference speed. Simultaneously, it combines Transformer and convolutional modules to enhance feature representation and cross-modal semantic interaction, ultimately generating more semantically consistent segmentation results. It is important to note that the skip connection mechanism in this embodiment is not part of the backbone network, encoder, or decoder. Its main function is to fuse raw information with deeply processed features, thereby preserving more detailed information and improving feature representation capabilities. Taking a lightweight backbone network as an example, its structure is primarily used for feature extraction, and the skip connection mechanism acts as an auxiliary module connecting these structures to enhance information flow between features at different levels.

[0080] The specific steps are as follows:

[0081] Step 2.2: Encode the training set and extract basic features.

[0082] In one specific implementation, during the encoding phase, the support image and the query image are input into a model trained on ImageNet to obtain feature representations at different levels. These different levels of features include low-level features, mid-level features, and high-level features. Specifically, in the scene segmentation task of autonomous driving, low-level features mainly include basic visual information such as edges, textures, and colors, which can help the model identify details such as road boundaries and lane lines. Mid-level features capture semantic components such as region structure and object contours, and are key to connecting low-level perception with high-level understanding, helping to locate the position and shape of targets such as vehicles and pedestrians. High-level features contain rich semantic information, enabling semantic understanding and recognition of different categories of objects, such as distinguishing traffic signs, traffic lights, pedestrians, and vehicles, improving the overall accuracy and generalization ability of segmentation. These three types of features are progressively layered and work synergistically to achieve refined semantic segmentation of complex traffic scenes.

[0083] The feature extraction process includes multiple stages: First, a low-level feature is obtained through a convolutional layer with a kernel size of 7×7 and a stride of 2, with an output size of 119×119×128; then, a feature map containing more spatial structure information is extracted in the intermediate feature extraction stage, with a size of 60×60×1024; finally, a high-level feature containing abstract semantic features is output, with a size of 60×60×2048.

[0084] Step 2.3: Use the scene segmentation model to perform dual-axis self-attention feature extraction on the image frames after semantic feature extraction.

[0085] In one specific implementation, three feature branches are used to enhance the features of the image frame from multiple aspects, and then the feature maps obtained from the three feature branches are fused to obtain fused features.

[0086] Specifically, this embodiment designs a BAFM module, such as... Figure 4 As shown, the three feature branches are the directional branch, the original branch, and the global branch. The directional branch extracts directional weights by performing pooling operations in both the X and Y directions, followed by one-dimensional convolution and softmax operations to generate directional weights. The original branch preserves basic texture information, which is then multiplied element-wise by the directional weights through convolution to enhance regions with significant directional features in the image. The global branch expands the receptive field through depthwise separable convolution to generate... Features are then merged using a self-attention mechanism to obtain global contextual information. Specifically, attention weights are obtained by calculating the similarity between Q and K, and then V is weighted and summed to generate fused features. The global contextual information extracted from the global branch is added to the output of the original branch to further enrich the overall representation of the feature map.

[0087] To support features For example, its formula is expressed as:

[0088] ,

[0089] ,

[0090] ,

[0091] ,

[0092] .

[0093] in, Indicates directional branch, This indicates the weights generated by the directional branches. This indicates the characteristics of the original branch. This indicates that the kernel size is 1. This indicates that the kernel size is 3. This represents the features generated by depthwise separable convolutional groups. Characteristics representing global branches This indicates the final enhanced feature.

[0094] Step 2.4: Use the scene segmentation model to perform information correction processing on the image frames after semantic feature extraction.

[0095] In one specific implementation, the fused features are fed into a channel correction unit and a spatial correction unit for correction, resulting in channel correction features and spatial correction features, respectively. The channel correction features and spatial correction features are then combined to obtain the final calibration features.

[0096] In autonomous driving scene segmentation tasks, merging textual and visual information helps to more fully understand semantic relationships in complex traffic environments. By fusing textual information such as scene descriptions, map annotations, or traffic rules with visual image features, the model's ability to recognize specific targets (such as bus lanes, speed limit signs, and construction zones) can be enhanced. For example, by using a visual language pre-trained model, regions in an image are aligned with corresponding text labels or descriptions to construct cross-modal feature representations, and then pixel-level fine semantic parsing is achieved through an attention mechanism. This multimodal fusion approach combines the intuitiveness of visual perception with prior knowledge of linguistic information, improving the model's understanding and segmentation accuracy of key targets in variable traffic scenarios. To demonstrate its specific adaptability to autonomous driving tasks, this embodiment corrects and aligns image and textual information from both channel and spatial dimensions, enabling a deeper exploration of semantic relationships within the traffic environment.

[0097] Specifically, such as Figure 5As shown, this embodiment designs an ICM module, which uses advanced query features. With text information As parallel inputs, they are fed into the channel correction unit and the spatial correction unit, respectively. The channel correction unit generates channel attention weights and applies them to the features by constructing the interaction relationship between image features and text semantics in the channel dimension. The spatial correction unit calculates a spatial attention map based on the spatial alignment information of the image and text and applies it to the spatial dimension of the image features. After the two units complete feature correction in their respective dimensions, the final output formula is:

[0098] ,

[0099] ,

[0100] .

[0101] in, and These represent the characteristics after passing through the Channel Correction Unit (CCU) and the Spatial Correction Unit (SCU), respectively, while Fw represents the calibration characteristics after passing through the ICM.

[0102] Step 2.5: Fuse the corrected multi-source features.

[0103] In one specific implementation, this embodiment designs an MFFM module. First, a spatial attention mechanism is applied to the calibration features to enhance the response of salient regions in the spatial dimension, highlighting the location information of key targets. Second, a cross-attention mechanism is introduced to interactively model the spatially enhanced image features with auxiliary features from other modalities or levels, achieving the fusion and reconstruction of multi-source semantic information through mutual guidance between features. Finally, the fused features are decoded to obtain the prediction mask. The formula is as follows:

[0104] ,

[0105] .

[0106] in, This represents the output of the deep multi-source feature fusion module, here satisfy Relationship, This represents the features of the symbiotic information after processing by the spatial attention module. This represents the features of the similarity vectors after processing by the spatial attention module. This represents the enhanced features of the AW attention module. This represents the enhanced features of the WA attention module. DecoderThis indicates two types of decoders. In this embodiment, they are a pixel decoder and a Transformer decoder. This represents the prediction mask.

[0107] Finally, the feature map is upsampled using bilinear interpolation to restore it to the same spatial resolution as the original image, generating a predicted output map with a size of 473×473×number of classes.

[0108] Step 3: Use the trained scene segmentation model to perform scene segmentation processing on the real-time video stream to be segmented.

[0109] Example 2:

[0110] Embodiment 2 of the present invention provides a scene segmentation system based on multimodal few-shot learning, comprising:

[0111] The data acquisition module is configured to acquire autonomous driving video streams, extract image frames from the video streams, and perform data augmentation processing on the image frames to obtain a dataset.

[0112] The model training module is configured to train and test the scene segmentation model using the dataset to obtain the trained scene segmentation model. The module uses a lightweight backbone network to extract semantic features from the dataset, then uses the scene segmentation model to perform dual-axis self-attention feature extraction and information correction on the image frames after semantic feature extraction, and finally fuses the corrected multi-source features to obtain the predicted output image.

[0113] The scene segmentation module is configured to use the trained scene segmentation model to perform scene segmentation processing on the real-time video stream to be segmented.

[0114] Example 3:

[0115] Embodiment 3 of the present invention provides a computer-readable storage medium storing a computer program adapted for loading by a processor and executing the steps in the scene segmentation method based on multimodal few-shot learning as described in Embodiment 1 of the present invention.

[0116] Example 4:

[0117] Embodiment 4 of the present invention provides a computer device, the device comprising:

[0118] A processor, adapted to execute computer programs;

[0119] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps in the scene segmentation method based on multimodal few-shot learning as described in Embodiment 1 of the present invention.

[0120] The steps and methods involved in Examples 2, 3 and 4 above correspond to those in Example 1. For specific implementation details, please refer to the relevant description section of Example 1.

[0121] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0122] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0123] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A scene segmentation method based on multi-modal small sample learning, characterized in that, The method comprises the following steps: An automatic driving video stream is acquired, image frames in the video stream are extracted, and data augmentation processing is performed on the image frames to obtain a data set; the specific steps of extracting the image frames in the video stream and performing data augmentation processing on the image frames are as follows: Image frames are extracted from video streams of different driving scenes at a fixed frame rate, and the image frames are labeled; The image frames are subjected to multi-strategy joint data enhancement processing; The enhanced image frames are subjected to normalization processing to obtain the data set, specifically, multi-strategy joint data enhancement operations are performed on image data in the automatic driving scene, including random cropping, mirror flipping, color disturbance, blurring, and noise superposition; The image frames are subjected to blurring processing, a Gaussian filter is called to perform convolution operation on the image matrix to smooth the image edges and reduce the intensity of the detailed texture; the corresponding label map remains unchanged to ensure semantic alignment; Artificial noise is superimposed in the image, Gaussian noise or salt and pepper noise is added to part of the pixels with a set probability, and the image pixel value is randomly disturbed to simulate perception anomalies or sensor interference; The above two enhancement methods are automatically combined and called at each training iteration to realize cross-enhancement of the data, and the blur kernel size and noise intensity range are determined according to a preset strategy, and different versions of the disturbed image are randomly generated each time the image is loaded; The data set is divided into a training set and a test set; The scene segmentation model is trained and tested using the data set to obtain the trained scene segmentation model, wherein a lightweight backbone network is used to extract semantic features from the data set, then the scene segmentation model is used to extract double-axis self-attention features and correct information from the image frames after semantic feature extraction, and finally the corrected multi-source features are fused to obtain a prediction output map; The trained scene segmentation model is used to perform scene segmentation processing on a real-time video stream to be segmented.

2. The scene segmentation method based on multi-modal small sample learning according to claim 1, characterized in that, The parameters of the lightweight backbone network have been pre-frozen.

3. The scene segmentation method based on multi-modal small sample learning according to claim 1, characterized in that, The specific steps of using the scene segmentation model to perform double-axis self-attention feature extraction processing on the image frames after semantic feature extraction are as follows: Three feature branches are used to enhance the features of the image frames from multiple aspects; The feature maps obtained by the three feature branches are fused to obtain fused features.

4. The scene segmentation method based on multi-modal small sample learning according to claim 3, characterized in that, The three feature branches are a direction branch, an original branch, and a global branch, the direction branch is used to extract directional weights, the original branch is used to retain basic texture information, and the global branch expands the receptive field through depth separable convolution.

5. The scene segmentation method based on multi-modal small sample learning according to claim 3, characterized in that, The specific steps of using the scene segmentation model to perform information correction processing on the image frames after semantic feature extraction are as follows: The fused features are sent to a channel correction unit and a spatial correction unit for correction to obtain channel correction features and spatial correction features, respectively; The channel correction features and the spatial correction features are combined to obtain final calibration features.

6. The scene segmentation method based on multi-modal small sample learning according to claim 5, characterized in that, The specific steps of fusing the corrected multi-source features are as follows: Spatial attention mechanisms are applied to the calibration features to strengthen the response of the salient regions from the spatial dimension and highlight the position information of the key targets; Cross-attention mechanisms are introduced to interactively model the spatially enhanced image features and auxiliary features from other modalities or levels, and to realize the fusion and reconstruction of multi-source semantic information through the mutual guidance between features. The fused features are finally output to obtain a prediction mask.

7. A scene segmentation system based on multi-modal few-shot learning, characterized in that, The method comprises the following steps: A data acquisition module is configured to acquire an automatic driving video stream, extract image frames in the video stream, and perform data augmentation processing on the image frames to obtain a data set. Image frames are extracted from video streams of different driving scenes at a fixed frame rate, and the image frames are labeled. The image frames are subjected to multi-strategy joint data enhancement processing. The enhanced image frames are subjected to normalization processing to obtain a data set. The image frames are subjected to blur processing, a Gaussian filter is called to perform convolution operation on the image matrix to smooth the image edges and reduce the detail texture intensity; the corresponding label map remains unchanged to ensure semantic alignment. Artificial noise is superimposed in the image, Gaussian noise or salt and pepper noise is added to part of the pixels with a set probability, and the image pixel value is randomly disturbed to simulate perception anomalies or sensor interference. The above two enhancement methods are automatically combined and called at each training iteration to realize cross enhancement of the data, and the blur kernel size and noise intensity range are determined according to a preset strategy, and different versions of the disturbed images are randomly generated each time the images are loaded. The data set is divided into a training set and a test set. A model training module is configured to train and test the scene segmentation model using the data set to obtain a trained scene segmentation model, wherein a lightweight backbone network is used to extract semantic features from the data set, then the scene segmentation model is used to extract double-axis self-attention features and information correction processing from the image frames after semantic feature extraction, and finally the corrected multi-source features are fused to obtain a prediction output map. A scene segmentation module is configured to use the trained scene segmentation model to perform scene segmentation processing on a real-time video stream to be segmented.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by the processor to implement the scene segmentation method based on multi-modal small sample learning according to any one of claims 1-6.

9. A computer device, characterized in that, a processor adapted to execute a computer program; a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by the processor, the scene segmentation method based on multi-modal small sample learning according to any one of claims 1-6 is realized.

Images