A scene recognition method and system based on semantic contrast visual transformation

Abstract

The application belongs to the field of scene recognition, and provides a scene recognition method and system based on semantic contrast visual transformation, which obtains corresponding branch feature tensors based on an RGB image and a double-branch network structure; wherein a first branch captures visual features through an RGB image and a visual feature extraction network, and a second branch obtains a semantic segmentation score map through an RGB image and a semantic segmentation network; a semantic converter is used to process the semantic segmentation score map, a convolution structure and a multi-head self-attention mechanism are inserted into the semantic converter, an inductive bias is introduced, and semantic features are extracted, wherein the semantic features include object category information and spatial layout within a scene; the visual features and the semantic features are compared and fused to obtain a fused scene comprehensive representation; scene recognition is performed according to the scene comprehensive representation to obtain a scene classification result. Significant improvement has been achieved in scene recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of scene recognition, and particularly relates to a scene recognition method and system based on semantic contrast visual transformation. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Scene recognition is a basic task in computer vision, which has attracted much attention due to its application in automatic driving and robotics. However, scene recognition faces more problems than general object recognition tasks due to the more abstract representation in scene images. In today's era, the increasing diversity of scene types increases the complexity of the recognition process. Common challenges in large-scale data sets include high inter-class similarity, scale and viewpoint changes, etc.

[0004] The inventors found that scene information is highly abstract and lacks obvious inter-class differences. Existing networks have difficulty in effectively extracting features from a single modality to accurately represent scenes. For example, images in libraries, bookstores and video stores have similar layouts. In addition, these three types of images also contain objects with similar shapes. Convolutional neural networks are good at local feature extraction, but are not good at establishing relationships between local regions and global texts. Therefore, CNNs have difficulty in fully considering relevant information in complex scene images.

[0005] Visual Transformers have made significant progress in recent years. Applying ImageNet pre-trained ViT for scene recognition has achieved gratifying results. However, this does not fully exploit the potential of visual transformers in scene recognition. To improve model performance, the typical method is to increase the depth and size of the network, such as BERT and GPT models. However, this method requires more hardware.

[0006] Multi-angle and multi-modal feature extraction and fusion are the mainstream methods to improve model capability. However, there are still challenges in achieving effective results in multi-modal learning. The original image modality and the semantic information modality require different feature extraction networks, but in order to complement and facilitate fusion, the feature flow is limited to similar or identical forms. SUMMARY

[0007] To solve at least one technical problem in the background art, the present application provides a scene recognition method and system based on semantic contrast visual transformation, which fully utilizes the powerful functions of visual transformer (ViT) and self-attention mechanism, combines RGB images and semantic information obtained through semantic segmentation, and provides comprehensive visual and contextual clues through SC-ViT to improve scene representation. Through a contrast learning framework, SC-ViT realizes effective cross-modal alignment by fusing feature representations from RGB and semantic modes, and improves the scene recognition performance of the model.

[0008] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0009] The first aspect of the present application provides a scene recognition method based on semantic contrast visual transformation, comprising the following steps:

[0010] Obtaining an RGB image;

[0011] Based on the RGB image and a double-branch network structure, a feature tensor corresponding to each branch is obtained; wherein, in the first branch, visual features are captured through the RGB image and a visual feature extraction network; in the second branch, a semantic segmentation score map is obtained through the RGB image and a semantic segmentation network; a semantic transformer is used to process the semantic segmentation score map; a convolution structure and a multi-head self-attention mechanism are inserted into the semantic transformer; an inductive bias is introduced; and semantic features are extracted, which include object class information and spatial layout within a scene.

[0012] The visual features and the semantic features are compared and fused to obtain a comprehensive scene representation after fusion;

[0013] Scene recognition is performed according to the comprehensive scene representation to obtain a scene classification result.

[0014] The second aspect of the present application provides a scene recognition system based on semantic contrast visual transformation, comprising:

[0015] An image acquisition module for acquiring an RGB image;

[0016] A multi-modal feature extraction module for obtaining a feature tensor corresponding to each branch based on the RGB image and a double-branch network structure; wherein, in the first branch, visual features are captured through the RGB image and a visual feature extraction network; in the second branch, a semantic segmentation score map is obtained through the RGB image and a semantic segmentation network; a semantic transformer is used to process the semantic segmentation score map; a convolution structure and a multi-head self-attention mechanism are inserted into the semantic transformer; an inductive bias is introduced; and semantic features are extracted, which include object class information and spatial layout within a scene.

[0017] a feature fusion module configured to compare and fuse the visual features and the semantic features to obtain a comprehensive scene representation after fusion;

[0018] a scene recognition module configured to recognize a scene according to the comprehensive scene representation to obtain a scene classification result.

[0019] A third aspect of the present application provides a computer-readable storage medium.

[0020] A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the scene recognition method based on semantic comparison and visual transformation according to the first aspect.

[0021] A fourth aspect of the present application provides a computer device.

[0022] A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the scene recognition method based on semantic comparison and visual transformation according to the first aspect when executing the program.

[0023] Compared with the prior art, the present application has the following beneficial effects:

[0024] 1. The present application uses an end-to-end trainable model for scene recognition, which can fuse multiple modalities to form more accurate scene information representation, and uses a semantic converter to process the semantic segmentation score map, inserts a convolution structure and a multi-head self-attention mechanism into the semantic converter, introduces an inductive bias, extracts semantic features, reduces the size of model parameters, and improves the recognition efficiency.

[0025] 2. In the multi-modal and multi-branch network, the model is trained by introducing a contrastive loss, the present application effectively aligns different modalities of the same sample and forms a cross-modal coherent representation. Not only can accurate recognition be achieved, but also each modality can be correctly aligned.

[0026] The advantages of the additional aspects of the present application will be partially given in the following description, partially become obvious from the following description, or be learned through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0027] The drawings constituting a part of the specification of the present application are used to provide further understanding of the present application, the illustrative embodiments of the present application and the description thereof are used to explain the present application, and do not constitute an improper limitation on the present application.

[0028] Figure 1 is the overall block diagram of the scene recognition method based on semantic comparison and visual transformation provided by the embodiments of the present application;

[0029] Figures 2(a)-2(b) Figure 2(a) is a structure of a semantic converter, and Figure 2(b) is a structure of a convolutional feed-forward network (CFFN), provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] The present application will be further described with reference to the drawings and embodiments.

[0031] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0032] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0033] As mentioned in the background, scene recognition is an important part of scene understanding, but it is a challenging task in image recognition. Despite the great progress made by deep learning, especially the emergence of convolutional neural networks (CNNs), scene recognition still faces some unresolved problems. This is mainly due to the high complexity of scene images, which cannot be correctly identified by simply recognizing a few objects in the image. In addition, the scene categories are numerous, and single-modal learning is prone to confusion.

[0034] The present application proposes an end-to-end multi-modal network SC-ViT, which is based on ViT and uses a powerful self-attention mechanism to capture visual and contextual clues from RGB images and semantic information. Semantic information is obtained through semantic segmentation, including object class information and spatial layout within the scene. Combining semantic information with RGB images forms a comprehensive representation of the scene. Specifically, two branches with different structures but equipped with self-attention mechanisms are used to extract features from RGB images and semantic information. Through a contrastive learning framework, SC-ViT unifies the feature representations of RGB and semantic modalities, enhancing their consistency and discriminability in expressing scenes. Experimental evaluations on the MIT Indoor67 and SUN397 datasets show that SC-ViT outperforms state-of-the-art methods and achieves significant improvements in scene recognition.

[0035] Embodiment One

[0036] As Figure 1 shown, the embodiment provides a scene recognition method based on semantic contrast visual transformation, comprising the following steps:

[0037] Step 1: Obtain an RGB image;

[0038] Step 2: Process the RGB image in the first branch, first flatten the image I∈R w×h×3 Through a patch embedding module, the image is divided into multiple patches, the patches are flattened, classification labels are added, and finally a tensor M is generated. M is sent to an L1 layer transformer encoder to obtain a feature tensor F I .

[0039] Step 3: Process the RGB image in the second branch, first pass the RGB image through a semantic segmentation module to generate a tensor S∈R w×h×f , where f represents the number of object categories. Similarly, the semantic tensor is patch-embedded to obtain a tensor E. E is encoded by a 5-layer dense connection semantic encoder. The weighted sum of the 5-layer network output is used as a feature tensor F s .

[0040] Step 4: Feature fusion, fuse the tensors F I and F s in the contrast module to obtain a final feature tensor F t .

[0041] Step 5: F t is input into an MLP (Multi-Layer Perceptron) through an L2 layer transformer encoder, and a classification result is obtained.

[0042] In order to more clearly illustrate the technical solutions of the present application, the following will be described in detail in the manner of specific embodiments.

[0043] In step 2, in this branch, the first L1 layer of the pre-trained ViT is integrated, and the remaining L2 layers are used as a decoder to form a complete ViT architecture. Thanks to the excellent performance of the pre-trained ViT, it is found that the L2 layer structure is very effective for extracting information from complex scene images. This design enhances the feature extraction capability without increasing the network size.

[0044] In step 3, in this branch, the embodiment uses ViT-Adapter as a frozen weight semantic segmentation network. The input image I∈R w×h×3 is processed by the pre-trained ViT-Adapter to generate a semantic segmentation score map S∈R w×h×F (where F is the number of object categories defined by ADE20k).

[0045] Further, the embodiment uses a semantic converter to process the semantic segmentation score map. Since the semantic information lacks texture information, the features are mainly concentrated in the class probability information and the local contour. The embodiment selects a convolution structure, specifically a convolution feedforward network (CFFN) instead of a feedforward network, and trains the network from scratch.

[0046] The structure of the semantic converter is shown in FIG. 2(a), and the specific processing process includes:

[0047] First, the semantic segmentation score map X is subjected to layer normalization processing, and then the normalized result is input into the convolution feedforward network. The output result is added with the first residual to obtain X cff ;

[0048] Then, X cff is subjected to layer normalization processing, and then the normalized result is input into the multi-head self attention mechanism (MHSA). The output result is added with the second residual X cff to obtain X out , which is the output of the semantic conversion block.

[0049] The above process can be expressed as:

[0050] X cff = X + CFFN(LayerNorm(X))

[0051] X out = X cff + MHSA(LayerMorm(X))

[0052] Wherein, X, X cff , X out are the input, output of the CFFN and output of the semantic conversion block, respectively.

[0053] In addition to the difference in the structure of the semantic converter, the method of the embodiment is another difference from the conventional converter block in the positions of the FFN and the MHSA.

[0054] The conventional method is to first pass the image through the self-attention mechanism, and then perform convolution processing on the obtained result. The embodiment exchanges the positions of the MHSA and the FFN to prevent the decomposition of the local features of the semantic information, thereby improving the recognition effect of the scene.

[0055] Specifically, the convolution feedforward network CFFN has the architecture shown in FIG. 2(b).

[0056] Since the embodiment needs to extract features through convolution, the embodiment reshapes the 1D label X into a 2D feature map X mUnlike traditional methods, the embodiment directly reshapes each one-dimensional mark into a two-dimensional map. This allows the extraction of salient features to be simple.

[0057] Then, the embodiment directly applies x m to a convolution operation:

[0058]

[0059] where Act is a nonlinear activation function, and DWConv denotes a depthwise convolution.

[0060] Existing MobileNet extracts features through the introduction of an inverted residual bottleneck block (IRB). However, the embodiment only selects depthwise convolution as the basic unit for two reasons. First, it helps to reduce the computational load; second, it is more consistent with the processing of semantic information by the transformer. In depthwise convolution, there is no interaction between channels, so a pointwise convolution is usually needed to cooperate with it. However, the self-attention mechanism involves the interaction of information from different positions. The success of the transformer proves the ability of this method of information interaction.

[0061] Therefore, the embodiment considers the feedforward network and the self-attention mechanism as a whole. The effectiveness of the scheme of the embodiment is verified in the ablation experiment of the embodiment.

[0062] Finally, in order to prepare for MSHA, the embodiment converts to 1D token:

[0063]

[0064] where Map2Token(-) is an operation that stretches the two-dimensional feature map to one dimension. C is the output of CFFN, which has the same shape as X.

[0065] In the embodiment, the multi-head self-attention mechanism is responsible for establishing relationships between different regions. Multi-head self-attention refers to the execution of the self-attention mechanism multiple times, so the embodiment focuses on the multi-head self-attention mechanism. It involves three intermediate tensors: query, key, and value. The embodiment denotes these tensors as Q, K, and V, respectively. The generation process of these three tensors is as follows:

[0066] (Q,K,V)=(X cff W q ,X cff W k ,X cff W v )

[0067] where X cff is the input, W q ,Wk W v denote trainable matrices of the fully connected layers for producing Q, K, V.

[0068] The dot product between the query and the key is computed, and then the softmax is applied to normalize the result, resulting in a score for each position. Multiplying this value by the score matrix gives the self-attention result. The calculation formula is as follows:

[0069]

[0070] where d k denotes the channel dimension of K, which can be regarded as a normalization parameter to ensure gradient stability.

[0071] In step 4, in the multi-modal and multi-branch network, it is crucial to train the model to not only achieve accurate recognition but also ensure that each modality is correctly aligned.

[0072] By introducing the contrastive loss, the embodiment can effectively align different modalities of the same sample and form a coherent representation across modalities. The calculation process of the contrastive loss can be represented as follows:

[0073]

[0074]

[0075] where x i and y j are the class labels of the image in the ith pair and the class labels of the semantic information in the jth pair, m i and n j are the feature embeddings of the two modal pairs. N is the batch size, and τ1 and τ2 are the temperatures of the scaled logarithm.

[0076] represents the loss generated in the class label alignment process, and corresponds to the loss related to the feature label.

[0077] The combination of the two loss functions is the contrastive loss which enables the embodiment to jointly adjust the scene feature vector and the class label.

[0078]

[0079] The total loss function of the model is composed of the contrastive loss and the classification loss .

[0080]

[0081] Unlike the cross-modal alignment method that considers the bidirectional alignment between text and image in the past, the embodiment studies the close relationship between image and semantic information, and only conducts unidirectional modal alignment from image to semantic.

[0082] Since the semantic information comes from the image, the embodiment finds that only aligning the image with the semantic modality is more suitable for the task of the embodiment.

[0083] The embodiment uses summation operation to fuse the information of the two alignment tensors.

[0084] F t =F I +F S

[0085] Where F I and F S are the alignment tensors from the image and the semantic information respectively. F t is the output of the contrast module. This operation selectively amplifies the response value of the information area while reducing the response value of the irrelevant area. After fusing the benign representations of the two modes, the fusion vector F t can more comprehensively and richly describe the scene, thereby improving the input quality of the subsequent classification task.

[0086] In order to verify the effectiveness of the present application, experiments are conducted, and the embodiment will evaluate the effectiveness of the SC-ViT model proposed in the embodiment on the widely used Massachusetts Institute of Technology Indoor67 and SUN397 data sets.

[0087] The embodiment outlines these data sets and shows the achievements made on these data sets. In addition, the embodiment also conducts several ablation experiments to gain a deeper understanding of the specific structure of SC-ViT.

[0088] Data sets

[0089] MIT Indoor67 contains 67 indoor scene categories, a total of 15,620 images. According to the standard evaluation protocol, the embodiment selects 100 images for each category for model training, of which 80 are used for training and 20 are used for testing.

[0090] SUN397 contains 397 scene categories, including outdoor and indoor scenes, totaling 108,754 images. Each category has at least 100 images. According to the original evaluation method, the embodiment selects 50 images for each category for training and 50 images for testing.

[0091] The embodiment adjusts the RGB image to 224x224 size as the input of SC-ViT.

[0092] Therefore, the token size generated by the network is 197x768, and the restored image size is 16x16. This embodiment selects 6 layers for the RGB branch and decoder. The semantic segmentation network is pre-trained using 150 object categories. The CFFN uses 3x3 depthwise convolutions followed by ReLU non-linear activation functions. In the comparison module, α and β are both set to 0.5, while λ... con and λ cls Set them to 1.0 and 1.0 respectively.

[0093] In this embodiment, the entire dual-branch network is trained as an end-to-end model.

[0094] The training parameters were set as follows: 30 training epochs, batch size of 32, initial learning rate of 0.0005, and decay every 10 epochs. For data augmentation, this embodiment used conventional random cropping, horizontal flipping, Gaussian blur, contrast normalization, Gaussian noise, and brightness variation on the RGB images. For semantic information, this embodiment only used horizontal flipping and conventional random cropping.

[0095] Table 1 Comparison of MIT Indoor67 and SUN397 datasets

[0096]

[0097] This invention evaluates SC-ViT on MIT Indoor67 and SUN397. These two datasets are commonly used datasets that contain various indoor and outdoor scenes, allowing for a comprehensive evaluation of the model's performance.

[0098] Table 1 lists the comparative data for the MIT Indoor67 and SUN397 datasets. The ViT baseline chosen in this invention outperforms previous models, such as hybrid CNNS and VSAD, in scene recognition tasks, but falls short of the performance achieved by newer CNN-based models. However, the SC-ViT proposed in this invention shows promising improvements over the baseline. On the MIT Indoor67 dataset, this embodiment achieves a 4.11% improvement, and on the SUN397 dataset, it achieves a 2.33% improvement. These results are superior to the comparative models. While existing techniques have also explored multimodal learning using semantic information, the feature extraction and fusion methods of this invention have proven to be more effective. MRNet employs a comprehensive and detailed feature extraction approach but does not offer significant improvements. In recent years, DPP-Net-101 has demonstrated impressive performance on these datasets. However, it should be noted that DPP-Net-101 was pre-trained on the Place365 Challenge dataset, which contains 8 million scene images, and then the classifier was further fine-tuned. In contrast, the method of this invention involves training the entire network from scratch, with only the RGB branch and the decoder utilizing widely used ImageNet-1K pre-trained weights as initial weights. Therefore, a direct comparison between the method of this invention and DPP-Net-101 may not be entirely meaningful. Overall, the comparative analysis in Table 1 validates the state-of-the-art performance of the proposed SC-ViT model in scene recognition tasks.

[0099] Ablation Research and Evaluation

[0100] This invention conducted several ablation experiments on the MIT Indoor67 dataset to analyze the effectiveness of different methods in integrating semantic information for multimodal learning. These experiments provide insights into the impact of various factors on model performance while maintaining consistency with the experimental settings described earlier.

[0101] Alignment between images and semantic information. Table 2 shows that unidirectional alignment from image to semantic information alone can lead to higher performance improvements. Furthermore, combining alignment with feature angles can further enhance model performance.

[0102] Table 1 compares the ablation results of different alignment methods in the loss function.

[0103]

[0104] Structural changes in the semantic transformer. Experiments in Table 3 show that more complex networks do not necessarily lead to better performance. Placing MHSA before CFFN for semantic feature extraction does not yield good results. This embodiment found that adding dense connections is beneficial for enhancing the representation of comprehensive information.

[0105] Table 2 Ablation results for different semantic converter architectures

[0106]

[0107]

[0108] This invention addresses the challenges of scene recognition by proposing the SC-ViT model, a novel multimodal network that fully leverages the power of visual transformers (ViTs) and self-attention mechanisms. By combining RGB images with semantic information obtained through semantic segmentation, SC-ViT provides comprehensive visual and contextual cues, thereby improving scene representation. Through a contrastive learning framework that fuses feature representations from RGB and semantic patterns, SC-ViT achieves effective cross-modal alignment, enhancing the model's scene recognition performance. Experimental results on the MIT Indoor67 and SUN397 datasets demonstrate this. SC-ViT outperforms state-of-the-art methods. This work not only advances the field of scene understanding but also reveals the potential of combining visual transformers with cross-modal alignment for a variety of multimodal learning applications.

[0109] Example 2

[0110] This embodiment provides a scene recognition system based on semantic contrast visual transformation, including:

[0111] Image acquisition module, which is used to acquire RGB images;

[0112] The multimodal feature extraction module is used to obtain the feature tensors of the corresponding branches based on RGB images and a dual-branch network structure. The first branch captures visual features through RGB images and a visual feature extraction network, and the second branch obtains a semantic segmentation score map through RGB images and a semantic segmentation network. The semantic segmentation score map is processed by a semantic converter. A convolutional structure and a multi-head self-attention mechanism are inserted into the semantic converter to introduce inductive bias and extract semantic features. The semantic features include object category information and spatial layout within the scene.

[0113] The feature fusion module is used to compare and fuse visual features and semantic features to obtain a fused comprehensive representation of the scene;

[0114] The scene recognition module is used to identify scenes based on comprehensive scene representations and obtain scene classification results.

[0115] Example 3

[0116] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a scene recognition method based on semantic contrast visual transformation as described in Embodiment 1.

[0117] Example 4

[0118] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the scene recognition method based on semantic contrast visual transformation as described in Embodiment 1.

[0119] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0120] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0124] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A scene recognition method based on semantic contrast visual transformation, characterized in that, Includes the following steps: Get an RGB image; Based on RGB images and a dual-branch network structure, feature tensors for the corresponding branches are obtained. The first branch captures visual features through RGB images and a visual feature extraction network. The second branch obtains a semantic segmentation score map through RGB images and a semantic segmentation network. A semantic converter is used to process the semantic segmentation score map. A convolutional structure and a multi-head self-attention mechanism are inserted into the semantic converter to introduce inductive bias and extract semantic features. The semantic features include object category information and spatial layout within the scene. Visual features and semantic features are compared and fused to obtain a fused comprehensive representation of the scene; the comparison and fusion of visual features and semantic features includes: by introducing contrast loss, the visual features and semantic features of the same sample are effectively aligned and a coherent representation across modalities is formed. The calculation process for the contrast loss is as follows: in, Represents the loss incurred during class tag alignment. This corresponds to the loss associated with the feature labels. and They are the first Class labeling and first class labeling of images Class labeling for semantic information in the middle, and These are the feature embeddings of two modality pairs. It refers to the batch size. 1 and 2 is the temperature scaled logarithm; Scene recognition is performed based on the comprehensive scene representation to obtain scene classification results.

2. The scene recognition method based on semantic contrast visual transformation as described in claim 1, characterized in that, The visual feature extraction network adopts the ViT architecture and uses ViT-Adapter as the semantic segmentation network with frozen weights.

3. The scene recognition method based on semantic contrast visual transformation as described in claim 1, characterized in that, The semantic segmentation score map is processed using a semantic converter. A convolutional structure and a multi-head self-attention mechanism are inserted into the semantic converter to introduce inductive bias and extract semantic features. Specifically, this includes: The semantic segmentation score map is subjected to layer normalization. The result of the normalization is input into the convolutional feedforward network. The output is added to the first residual to obtain the first semantic feature. The first semantic feature is subjected to layer normalization, and then the result of the normalization is input into the multi-head self-attention mechanism. The output is added to the second residual to obtain the second semantic feature.

4. The scene recognition method based on semantic contrast visual transformation as described in claim 3, characterized in that, In a convolutional feedforward network, each one-dimensional label in the semantic segmentation score map is first reshaped into a two-dimensional feature map. Using depthwise convolution as the basic unit, convolution operation is performed on the two-dimensional feature map. Then, the result of the convolution is stretched to a one-dimensional space and input into the multi-head self-attention mechanism.

5. The scene recognition method based on semantic contrast visual transformation as described in claim 1, characterized in that, In the multi-head self-attention mechanism, the training matrix of the fully connected layer and the output of the convolutional structure are used to generate the multi-head self-attention mechanism, which involves three intermediate tensors: query, key, and value. The score of each position is obtained by the dot product between the query and the key, and the self-attention result is obtained by multiplying the corresponding value and the score matrix.

6. A scene recognition system based on semantic contrast visual transformation, characterized in that, include: Image acquisition module, which is used to acquire RGB images; The multimodal feature extraction module is used to obtain the feature tensors of the corresponding branches based on RGB images and a dual-branch network structure. The first branch captures visual features through RGB images and a visual feature extraction network, and the second branch obtains a semantic segmentation score map through RGB images and a semantic segmentation network. The semantic segmentation score map is processed by a semantic converter. A convolutional structure and a multi-head self-attention mechanism are inserted into the semantic converter to introduce inductive bias and extract semantic features. The semantic features include object category information and spatial layout within the scene. The feature fusion module is used to compare and fuse visual features and semantic features to obtain a fused comprehensive scene representation; the comparison and fusion of visual features and semantic features includes: by introducing contrast loss, effectively aligning the visual features and semantic features of the same sample, and forming a coherent representation across modalities. The calculation process for the contrast loss is as follows: in, Represents the loss incurred during class tag alignment. This corresponds to the loss associated with the feature labels. and They are the first Class labeling and first class labeling of images Class labeling for semantic information in the middle, and These are the feature embeddings of two modality pairs. It refers to the batch size. 1 and 2 is the temperature scaled logarithm; The scene recognition module is used to identify scenes based on comprehensive scene representations and obtain scene classification results.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the scene recognition method based on semantic contrast visual transformation as described in any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the scene recognition method based on semantic contrast visual transformation as described in any one of claims 1-5.

Images