Scene segmentation method and system based on multi-modal small sample learning

Abstract

The invention discloses a scene segmentation method and system based on multi-modal small sample learning, and relates to the technical field of computer vision. The method comprises the following steps: acquiring an automatic driving video stream, and extracting image frames in the video stream; training and testing a scene segmentation model by using the data set to obtain a trained scene segmentation model, performing semantic feature extraction on the data set by using a lightweight backbone network, and then performing double-axis self-attention feature extraction and information correction processing on the image frame after semantic feature extraction by using the scene segmentation model to obtain a trained scene segmentation model; and finally, fusing the corrected multi-source features to obtain a prediction output graph. And performing scene segmentation processing on a real-time video stream to be segmented by using the trained scene segmentation model. According to the method, the understanding capability of semantic information in a complex scene can be effectively enhanced while the lightweight of the model is guaranteed, and the segmentation precision and generalization capability under the small sample condition are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a scene segmentation method and system based on multimodal small sample learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence and deep learning technologies, scene segmentation, a key task in image semantic understanding, has been widely used in fields such as autonomous driving, intelligent security, human-computer interaction, assisted diagnosis, and robotic perception. Its goal is to accurately classify each pixel in an image into its corresponding semantic category, thereby achieving a structured understanding of complex environments.

[0004] Traditional scene segmentation methods mostly rely on single-modal visual information, typically employing architectures such as convolutional neural networks (CNNs) or visual transformers to perform pixel-level classification on RGB images. While these methods have achieved promising results on standard datasets, they face numerous challenges in practical applications. For example, factors such as illumination variations, occlusions, image blur, and similar visual features between categories often reduce the robustness of single-modal methods, limiting their generalizability.

[0005] To improve the generalization and semantic understanding of segmentation models in real-world environments, researchers have gradually introduced multimodal information fusion strategies, specifically combining textual information such as semantic labels, image descriptions, and task instructions with visual information. This allows the model to not only "see" but also "understand." By leveraging the contextual information provided by natural language, the model can more accurately understand the semantic boundaries of objects in an image, improving classification accuracy. For example, when identifying a "walking person" versus a "standing person" in an image, the language description can provide cues about the action or state, thereby assisting the model in achieving more refined semantic segmentation.

[0006] However, the implementation of multimodal methods usually relies on large-scale labeled samples for model training, such as image-text pairs, image segmentation masks, etc. This high data dependence presents a significant bottleneck in practical applications: obtaining large-scale, high-quality multimodal datasets is expensive, and the labeling process is complex and time-consuming, especially in fine-grained tasks or specific fields such as autonomous driving, medical images, and remote sensing images. In autonomous driving scenarios, high-precision image segmentation and semantic understanding are particularly needed, which requires obtaining a large number of finely labeled images and corresponding environmental information. However, the labeling cost of autonomous driving data is extremely high, and environmental changes, such as weather, lighting, and traffic density, lead to the diversity of data distribution, making the collection and processing of large-scale labeled data more challenging. Therefore, how to train efficiently under small sample conditions has become an important direction of current research.

[0007] In autonomous driving, real-time performance is crucial, requiring the system to rapidly analyze and respond to road conditions. Therefore, lightweight and efficient models are essential. However, current mainstream multimodal scene segmentation networks generally have complex structures, large number of model parameters, and high computational resource consumption during training and inference. This makes them difficult to meet the requirements of edge computing, mobile deployment, and other application scenarios that require high real-time performance and resource-friendliness.

[0008] In summary, how to overcome the defects of existing multimodal scene segmentation methods, such as unsatisfactory small sample training effect, poor real-time performance, and low resource utilization, so that they can be efficiently applied in complex environments such as autonomous driving, has become an urgent problem to be solved in existing technologies. Summary of the Invention

[0009] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a scene segmentation method and system based on multimodal small sample learning, which has the advantages of lightweight, high precision, and strong generalization ability, and is suitable for resource-sensitive terminal devices and semantic understanding tasks in complex environments.

[0010] In order to achieve the above object, the present invention is implemented through the following technical solutions: A first aspect of the present invention provides a scene segmentation method based on multimodal small sample learning, comprising the following steps: Obtain the autonomous driving video stream, extract the image frames in the video stream, and perform data expansion processing on the image frames to obtain a data set; The scene segmentation model is trained and tested using the dataset to obtain a trained scene segmentation model. A lightweight backbone network is used to extract semantic features from the dataset. The scene segmentation model is then used to perform dual-axis self-attention feature extraction and information correction on the image frames after semantic feature extraction. Finally, the corrected multi-source features are fused to obtain a predicted output map. The trained scene segmentation model is used to perform scene segmentation processing on the real-time video stream to be segmented.

[0011] Furthermore, the specific steps of extracting image frames from the video stream and performing data expansion processing on the image frames are as follows: Extract image frames from video streams of different driving scenarios at a fixed frame rate and annotate the image frames; Perform multi-strategy data enhancement processing on image frames; Normalize the enhanced image frames to obtain a data set; Divide the dataset into training and testing sets.

[0012] Furthermore, the lightweight backbone network parameters have been frozen in advance.

[0013] Furthermore, the specific steps of using the scene segmentation model to perform dual-axis self-attention feature extraction on the image frame after semantic feature extraction are as follows: Use three feature branches to enhance the image frame from multiple aspects; The feature maps obtained by the three feature branches are fused to obtain the fused features.

[0014] Furthermore, the three feature branches are directional branch, primitive branch and global branch. The directional branch is used to extract directional weights, the primitive branch is used to retain basic texture information, and the global branch expands the receptive field through depth-wise separable convolution.

[0015] Furthermore, the specific steps of using the scene segmentation model to perform information correction processing on the image frame after semantic feature extraction are as follows: The fused features are sent to the channel correction unit and the spatial correction unit for correction to obtain channel correction features and spatial correction features respectively; The channel correction features and spatial correction features are combined to obtain the final calibration features.

[0016] Furthermore, the specific steps for fusing the corrected multi-source features are as follows: Applying a spatial attention mechanism to the calibration features to strengthen the response of the salient areas in the spatial dimension and highlight the location information of key targets; The cross-attention mechanism is introduced to interactively model the spatially enhanced image features with auxiliary features from other modalities or layers, and to achieve the fusion and reconstruction of multi-source semantic information through mutual guidance between features. Finally, the fused features are output for decoding to obtain the predicted mask.

[0017] A second aspect of the present invention provides a scene segmentation system based on multimodal small sample learning, comprising: a data acquisition module configured to acquire an autonomous driving video stream, extract image frames from the video stream, and perform data expansion processing on the image frames to obtain a data set; a model training module configured to train and test the scene segmentation model using the dataset to obtain a trained scene segmentation model, wherein a lightweight backbone network is used to extract semantic features from the dataset, and then the scene segmentation model is used to perform dual-axis self-attention feature extraction and information correction processing on the image frames after the semantic feature extraction, and finally the corrected multi-source features are fused to obtain a predicted output map; The scene segmentation module is configured to perform scene segmentation processing on the real-time video stream to be segmented using the trained scene segmentation model.

[0018] The third aspect of the present invention provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps in the scene segmentation method based on multimodal small sample learning as described in the first aspect of the present invention.

[0019] A fourth aspect of the present invention provides a computer device, comprising: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by the processor, the scene segmentation method based on multimodal small sample learning as described in the first aspect of the present invention is implemented.

[0020] One or more of the above technical solutions have the following beneficial effects: The present invention discloses a scene segmentation method and system based on multimodal small sample learning. In order to solve the problems of low accuracy of existing scene segmentation methods under small sample conditions, large models that are difficult to deploy, and insufficient fusion of multimodal information, a small sample scene segmentation method that integrates visual and textual information is proposed. The method has the advantages of being lightweight, highly accurate, and having strong generalization capabilities, and is suitable for resource-sensitive terminal devices and semantic understanding tasks in complex environments. The present invention realizes joint modeling of image and semantic information by extracting key frames from on-board videos and combining sensor data with natural language descriptions to construct image-text pairs. In the data preprocessing stage, a multi-strategy data enhancement and pixel normalization method is adopted to effectively expand sample diversity and alleviate the overfitting problem under small sample conditions. In terms of model design, the encoder is first used to extract image features and feed them into the dual-axis self-attention feature module (BAFM), which models contextual information from the horizontal, vertical and global levels respectively, and forms more recognizable deep features after fusion; then the information correction module (ICM) guides the semantic alignment of enhanced features and text information in the spatial and channel dimensions to improve the consistency and accuracy of cross-modal fusion; then, the features enter the multi-source feature fusion module (MFFM), which sequentially introduces spatial attention and cross-attention mechanisms to achieve full fusion of features of different scales and modalities, enhancing the perception of fine-grained targets. Finally, a decoder combined with the Transformer structure is used to model global dependencies, and pixel-level branches are used to restore image boundaries and details. The output segmented image is restored to its original size through bilinear interpolation to ensure that the results have clear edges and consistent semantics. While ensuring the lightweight model, the present invention effectively improves the segmentation accuracy and generalization ability in complex scenarios.

[0021] In order to solve the problem of limited number of training samples, this paper constructs a data preprocessing mechanism to enhance perception ability, expand the data set, increase data diversity, and prevent overfitting of small samples. This paper proposes a segmentation network composed of a visual-linguistic multimodal collaborative modeling mechanism. The network first uses a backbone network to extract multi-scale features of the support image and query image, respectively, and combines them with a support mask to enhance significant regions. The image features are then input to mine regional associations. Text information is mapped into low-dimensional semantic vectors through Word2Vec vector embedding, achieving feature alignment between the language and visual modalities and improving semantic consistency. Finally, the system spatially integrates image structural features and language semantic information, outputting a final predicted image with semantic information, completing the segmentation process of the entire system.

[0022] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 This is a flow chart of a scene segmentation method based on multimodal small sample learning in an embodiment of the present invention; Figure 2 Schematic diagram of the image preprocessing method in embodiment 1 of the present invention; Figure 3 Schematic diagram of a network framework based on multimodal small sample learning in Example 1 of the present invention; Figure 4 1 is a schematic diagram of a BAFM in Example 1 of the present invention; Figure 5 This is a schematic diagram of the ICM in the first embodiment of the present invention; Figure 6 Schematic diagram of MFFM in embodiment 1 of the present invention. DETAILED DESCRIPTION

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0026] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations; The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] Example 1: The first embodiment of the present invention provides a scene segmentation method based on multimodal small sample learning, such as Figure 1 As shown, the following steps are included: Step 1: Obtain the autonomous driving video stream, extract the image frames in the video stream, and perform data expansion processing on the image frames to obtain the dataset.

[0028] Step 1.1: Collect autonomous driving video streams from the autonomous driving scene.

[0029] Step 1.2: Extract image frames from the video stream and perform data augmentation on the image frames.

[0030] Step 1.2.1: Extract image frames from the video streams of different driving scenes at a fixed frame rate and annotate the image frames.

[0031] In one specific implementation, an autonomous driving video stream is first received and key image frames are extracted from different driving scenes at a fixed frame rate. Image-text pairs are then automatically constructed, combining vehicle sensor data and natural language descriptions to enhance multimodal information for scene understanding. Specifically, frame-by-frame semantic annotation is performed using a combination of manual or auxiliary annotation tools, assigning clear semantic category labels to each image frame to construct high-quality training and test datasets. Since videos consist of continuous image frames, an image can be extracted at fixed intervals (e.g., every 10 frames) to convert the video sequence into static images for use as model input.

[0032] In the image annotation stage, each pixel position is assigned a corresponding category label and represented in a specific digital encoding method, and finally a binary mask image corresponding to the image is generated to guide the model to learn the semantic structure in the image.

[0033] Step 1.2.2: Perform multi-strategy data enhancement processing on the image frame.

[0034] In a specific embodiment, a multi-strategy data enhancement operation is performed on the image data in the autonomous driving scenario, including but not limited to basic enhancement methods such as random cropping, mirror flipping, color perturbation, blurring, and noise superposition, to improve the robustness of the model under various environmental changes (such as different weather and lighting conditions) and alleviate the overfitting problem caused by small samples.

[0035] Specifically, such as Figure 2 As shown in the figure, to improve the model's generalization ability under limited sample conditions, various enhancement strategies are introduced in the constructed dataset to process the input images and their annotations, thereby expanding the data scale and effectively reducing the model's false positive rate. Considering the labor cost and annotation errors caused by pixel-by-pixel annotation, combining image enhancement and preprocessing techniques can significantly alleviate this problem, improving network performance while reducing reliance on massive amounts of data.

[0036] First, the image frame is blurred. Specifically, a Gaussian filter is used to perform a convolution operation on the image matrix to smooth image edges and reduce the intensity of detailed textures. The corresponding annotation map remains unchanged to ensure semantic alignment.

[0037] Then, artificial noise is superimposed on the image. Specifically, Gaussian noise or salt and pepper noise is added to some pixels with a set probability, simulating perceptual anomalies or sensor interference by randomly perturbing the image pixel values.

[0038] The above two enhancement methods are automatically combined and called in each training iteration to achieve cross-enhancement of the data. The blur kernel size and noise intensity range are determined according to the preset strategy. Different versions of the perturbed image are randomly generated when the image is loaded in each round.

[0039] Step 1.2.3: Normalize the enhanced image frame.

[0040] In a specific implementation, the image pixels are normalized and the pixel values ​​are mapped to a standardized interval to eliminate the distribution drift caused by the enhancement operation, ensuring that the image input is adapted to the subsequent neural network, thereby improving the accuracy and stability of the model in complex driving scenarios.

[0041] Step 1.2.4: Divide the dataset into training and testing sets.

[0042] Step 2: Use the dataset to train and test the scene segmentation model to obtain the trained scene segmentation model.

[0043] The lightweight backbone network is used to extract semantic features from the dataset. The scene segmentation model is then used to perform dual-axis self-attention feature extraction and information correction on the image frames after semantic feature extraction. Finally, the corrected multi-source features are fused to obtain the predicted output image. The specific steps are as follows: Step 2.1: Use the lightweight backbone network to extract semantic features from the dataset. To avoid overfitting during training, the lightweight backbone network parameters are frozen in advance and no gradient updates are performed to stabilize the overall feature expression structure.

[0044] In one specific embodiment, the scene segmentation model uses ResNet50 as the backbone network. The backbone network extracts features from the input image, and the model is built with the parameters of the frozen backbone model as the background. The enhanced image is then fed into a lightweight backbone network to extract multi-scale semantic features. A parameter freezing strategy is used to maintain the stability of low-level features, and the model is divided into a support set and a query set for training.

[0045] In the task of autonomous driving scene segmentation, semantic features generally refer to the semantic category information to which each pixel in the image belongs, such as the feature information reflected by elements such as roads, vehicles, pedestrians, buildings, traffic signs, and the sky. These features are automatically extracted from images through deep neural networks. They contain both spatial location information and high-level semantic understanding capabilities, which helps the model accurately identify and distinguish different types of targets and achieve comprehensive perception and understanding of complex traffic environments. This embodiment implements multi-scale extraction operations of semantic features by setting a backbone network with frozen parameters. It forms a multi-scale feature expression by retaining details at a shallow level and extracting semantics at a deep level, providing a strong data foundation for the data processing of subsequent scene segmentation models.

[0046] After a lightweight backbone network preprocesses the dataset to extract semantic features, the training set of road scene images is divided into a support set and a query set and input into the scene segmentation model for training. To eliminate irrelevant background interference, a Transformer structure and convolution operator are introduced to fuse image features and strengthen the complementary expression of local and global semantic information. This ensures both speed improvement and effective improvement in accuracy.

[0047] Specifically, the preprocessed road scene image training set is input Figure 3 The network structure shown is used for training. The scene segmentation model in this embodiment combines a Transformer architecture with a convolutional operator. During training, the encoder first extracts semantic features from the image and feeds them into the Bi-Axial Feature Attention Module (BAFM) to form a context-rich feature representation. Subsequently, the Information Calibration Module (ICM) spatially and semantically aligns the fused features to improve feature consistency. Furthermore, the features are fed into the Multi-Source Feature Fusion Module (MFFM). This module uses a multi-branch mechanism to fuse co-occurring information and similarity vectors, effectively alleviating the uneven distribution of multimodal information. It further integrates features at different scales and improves the perception of small objects. Finally, the Transformer decoder models global dependencies, while combining with a pixel-level decoder to restore image edges and details, jointly outputting the final segmentation prediction result. This design effectively improves the accuracy and boundary clarity of semantic segmentation in autonomous driving scenarios while ensuring real-time performance.

[0048] In terms of network structure, this embodiment innovatively introduces a skip connection mechanism to achieve multi-layer semantic and spatial information fusion between the encoder and decoder, effectively improving the accuracy and reasoning speed of the model, while combining the Transformer and convolution modules to enhance feature expression and cross-modal semantic interaction, and ultimately generating more semantically consistent segmentation results. It should be noted that the skip connection mechanism in this embodiment does not belong to the backbone network, encoder and decoder parts. Its main function is to fuse the original information with the features after deep processing, thereby retaining more detailed information and improving feature expression capabilities. Taking the lightweight backbone network as an example, its structure is mainly used for feature extraction, and the skip connection mechanism is connected to these structures as an auxiliary module to enhance the information flow between features at different levels.

[0049] The specific steps are as follows: Step 2.2: Encode the training set and extract basic features.

[0050] In a specific embodiment, during the encoding phase, the support image and the query image are input into a model that has been trained on ImageNet to obtain feature representations at different levels. Among them, features at different levels include low-level features, mid-level features, and high-level features. Specifically, in the scene segmentation task of autonomous driving, low-level features mainly include basic visual information such as edges, textures, and colors, which can help the model identify details such as road boundaries and lane lines. Mid-level features capture semantic components such as regional structure and object contours, and are the key to connecting low-level perception with high-level understanding, helping to locate the position and shape of targets such as vehicles and pedestrians. High-level features contain rich semantic information and can achieve semantic understanding and recognition of objects of different categories, such as distinguishing traffic signs, traffic lights, pedestrians, vehicles, etc., to improve the accuracy and generalization ability of overall segmentation. These three types of features are progressive and work together to achieve refined semantic segmentation of complex traffic scenes.

[0051] The feature extraction process includes multiple stages: first, a convolutional layer with a kernel size of 7×7 and a stride of 2 is used to obtain low-level features, with an output size of 119×119×128; then, a feature map containing more spatial structure information is extracted from the intermediate features, with a size of 60×60×1024; finally, high-level features containing abstract semantic features are output, with a size of 60×60×2048.

[0052] Step 2.3: Use the scene segmentation model to perform dual-axis self-attention feature extraction on the image frames after semantic feature extraction.

[0053] In a specific implementation, three feature branches are used to enhance features of an image frame from multiple aspects, and then the feature maps obtained by the three feature branches are fused to obtain fused features.

[0054] Specifically, this embodiment designs a BAFM module, such as Figure 4 As shown in the figure, the three feature branches are directional branch, original branch and global branch. The directional branch is used to extract directional weights, performs pooling operations in both X and Y directions, and then performs one-dimensional convolution and Softmax operations to generate directional weights. The original branch is used to retain basic texture information, which is then multiplied element-by-element by convolution with directional weights to enhance the areas with significant directional features in the image. The global branch expands the receptive field through depthwise separable convolution to generate The features are then merged through a self-attention mechanism to obtain global context. Specifically, attention weights are obtained by calculating the similarity between Q and K, and then weighted summed with V to generate the fused features. The global context information extracted by the global branch is added to the output of the original branch to further enrich the overall expression of the feature map.

[0055] To support features For example, the formula is: , , , , .

[0056] in, Indicates the direction branch, Indicates the weight of direction branch generation. Represents the feature of the original branch. Indicates that the convolution kernel size is 1, Indicates that the convolution kernel size is 3, represents the features generated by the depth-wise separable convolution group, represents the characteristics of the global branch, Represents the final enhanced features.

[0057] Step 2.4: Use the scene segmentation model to perform information correction processing on the image frame after semantic feature extraction.

[0058] In a specific embodiment, the fused features are sent to a channel correction unit and a spatial correction unit for correction to obtain channel correction features and spatial correction features, respectively. The channel correction features and spatial correction features are combined to obtain the final calibration features.

[0059] In autonomous driving scene segmentation tasks, combining textual and visual information helps to more fully understand semantic relationships within complex traffic environments. By integrating textual information such as scene descriptions, map annotations, or traffic regulations with visual image features, the model's ability to recognize specific objects (such as bus lanes, speed limit signs, and construction zones) can be enhanced. For example, using a pre-trained visual language model, image regions are aligned with corresponding textual labels or descriptions to construct a cross-modal feature representation. This attention mechanism then enables pixel-level, granular semantic parsing. This multimodal fusion approach combines the intuitiveness of visual perception with prior knowledge of language information to enhance the model's understanding and segmentation accuracy of key objects in highly variable traffic scenarios. To demonstrate specific adaptability to autonomous driving tasks, this embodiment aligns and aligns image and textual information along both channel and spatial dimensions, enabling deeper exploration of semantic relationships within the traffic environment.

[0060] Specifically, such as Figure 5 As shown, this embodiment designs an ICM module, which uses advanced query features With text message As parallel inputs, they are fed into the channel correction unit and the spatial correction unit, respectively. The channel correction unit generates channel attention weights and weights the features by constructing the interactive relationship between image features and text semantics in the channel dimension. The spatial correction unit calculates the spatial attention map based on the spatial alignment information of the image and text and acts on the spatial dimension of the image features. After the two units complete feature correction in their respective dimensions, the final output formula is: , , .

[0061] in, and They represent the features after the channel correction unit CCU and the spatial correction unit SCU respectively, and Fw represents the calibrated features after the ICM.

[0062] Step 2.5: Fuse the corrected multi-source features.

[0063] In a specific implementation, this embodiment designs an MFFM module. First, a spatial attention mechanism is applied to the calibration features to enhance the response of salient areas in the spatial dimension and highlight the location information of key targets. Second, a cross-attention mechanism is introduced to interactively model the spatially enhanced image features with auxiliary features from other modalities or levels. Through mutual guidance between features, the fusion and reconstruction of multi-source semantic information is achieved. Finally, the fused features are output and decoded to obtain the prediction mask. The formula is as follows: , .

[0064] in, Represents the output of the deep multi-source feature fusion module, where satisfy relationship, represents the features of the symbiotic information after being processed by the spatial attention module, Represents the features of the similarity vector after being processed by the spatial attention module, represents the features enhanced by the AW attention module, Represents the features after the WA attention module is enhanced. Decoder Indicates two decoders. In this embodiment, they are a pixel decoder and a Transformer decoder. Represents the prediction mask.

[0065] Finally, the feature map is upsampled by bilinear interpolation to restore it to the same spatial resolution as the original image, generating a prediction output map of size 473×473×number of categories.

[0066] Step 3: Use the trained scene segmentation model to perform scene segmentation processing on the real-time video stream to be segmented.

[0067] Example 2: A second embodiment of the present invention provides a scene segmentation system based on multimodal small sample learning, including: a data acquisition module configured to acquire an autonomous driving video stream, extract image frames from the video stream, and perform data expansion processing on the image frames to obtain a data set; a model training module configured to train and test the scene segmentation model using the dataset to obtain a trained scene segmentation model, wherein a lightweight backbone network is used to extract semantic features from the dataset, and then the scene segmentation model is used to perform dual-axis self-attention feature extraction and information correction processing on the image frames after the semantic feature extraction, and finally the corrected multi-source features are fused to obtain a predicted output map; The scene segmentation module is configured to perform scene segmentation processing on the real-time video stream to be segmented using the trained scene segmentation model.

[0068] Example 3: Embodiment 3 of the present invention provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor and executing the steps in the scene segmentation method based on multimodal small sample learning as described in embodiment 1 of the present invention.

[0069] Example 4: A fourth embodiment of the present invention provides a computer device, comprising: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by the processor, the steps in the scene segmentation method based on multimodal small sample learning as described in the first embodiment of the present invention are implemented.

[0070] The steps involved in the above embodiments 2, 3 and 4 correspond to those in the method embodiment 1. For the specific implementation methods, please refer to the relevant description part of the embodiment 1.

[0071] Those skilled in the art will appreciate that the units and algorithmic steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data processing device such as a server or data center that integrates one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)). The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technical object of a person skilled in the art that can be easily conceived of within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A scene segmentation method based on multimodal small sample learning, characterized in that: The following steps are involved: Obtain the autonomous driving video stream, extract the image frames in the video stream, and perform data expansion processing on the image frames to obtain a data set; The scene segmentation model is trained and tested using the dataset to obtain a trained scene segmentation model. A lightweight backbone network is used to extract semantic features from the dataset. The scene segmentation model is then used to perform dual-axis self-attention feature extraction and information correction on the image frames after semantic feature extraction. Finally, the corrected multi-source features are fused to obtain a predicted output map. The trained scene segmentation model is used to perform scene segmentation processing on the real-time video stream to be segmented.

2. The scene segmentation method based on multimodal small sample learning according to claim 1, characterized in that: The specific steps for extracting image frames from a video stream and performing data expansion processing on the image frames are as follows: Extract image frames from video streams of different driving scenarios at a fixed frame rate and annotate the image frames; Perform multi-strategy data enhancement processing on image frames; Normalize the enhanced image frames to obtain a data set; Divide the dataset into training and testing sets.

3. The scene segmentation method based on multimodal small sample learning according to claim 1, characterized in that: The lightweight backbone network parameters are frozen in advance.

4. The scene segmentation method based on multimodal small sample learning according to claim 1, characterized in that: The specific steps of using the scene segmentation model to perform dual-axis self-attention feature extraction on the image frame after semantic feature extraction are as follows: Use three feature branches to enhance the image frame from multiple aspects; The feature maps obtained by the three feature branches are fused to obtain the fused features.

5. The scene segmentation method based on multimodal small sample learning according to claim 4, characterized in that: The three feature branches are directional branch, primitive branch and global branch. The directional branch is used to extract directional weights, the primitive branch is used to retain basic texture information, and the global branch expands the receptive field through depth-wise separable convolution.

6. The scene segmentation method based on multimodal small sample learning according to claim 4, characterized in that: The specific steps of using the scene segmentation model to perform information correction processing on the image frame after semantic feature extraction are as follows: The fused features are sent to the channel correction unit and the spatial correction unit for correction to obtain channel correction features and spatial correction features respectively; The channel correction features and spatial correction features are combined to obtain the final calibration features.

7. The scene segmentation method based on multimodal small sample learning according to claim 6, characterized in that: The specific steps for fusing the corrected multi-source features are: Applying a spatial attention mechanism to the calibration features to strengthen the response of the salient areas in the spatial dimension and highlight the location information of key targets; The cross-attention mechanism is introduced to interactively model the spatially enhanced image features with auxiliary features from other modalities or layers, and to achieve the fusion and reconstruction of multi-source semantic information through mutual guidance between features. Finally, the fused features are output for decoding to obtain the predicted mask.

8. A scene segmentation system based on multimodal small sample learning, characterized by: include: a data acquisition module configured to acquire an autonomous driving video stream, extract image frames from the video stream, and perform data expansion processing on the image frames to obtain a data set; a model training module configured to train and test the scene segmentation model using the dataset to obtain a trained scene segmentation model, wherein a lightweight backbone network is used to extract semantic features from the dataset, and then the scene segmentation model is used to perform dual-axis self-attention feature extraction and information correction processing on the image frames after the semantic feature extraction, and finally the corrected multi-source features are fused to obtain a predicted output map; The scene segmentation module is configured to perform scene segmentation processing on the real-time video stream to be segmented using the trained scene segmentation model.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the scene segmentation method based on multimodal small sample learning according to any one of claims 1 to 7.

10. A computer device, characterized in that: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by the processor, implements the scene segmentation method based on multimodal small sample learning as described in any one of claims 1 to 7.

Images