A real-time semantic segmentation method and system with dual-resolution interactive attention

Abstract

The application discloses a kind of dual-resolution interactive attention real-time semantic segmentation method and system, comprising: obtaining the image information to be segmented under the setting scene;The image information is preprocessed and input into the dual-resolution interactive attention real-time semantic segmentation network model trained, to obtain image semantic segmentation result;Wherein, dual-resolution interactive attention real-time semantic segmentation network model first obtains the shallow feature map of image, shallow feature map is input into parallel high-resolution branch and low-resolution branch respectively, and bilateral information fusion is carried out between two branches, and the spatial feature and context feature of image are obtained respectively;The spatial feature and context feature obtained are summed by point, and the final image semantic segmentation result is obtained by decoding.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a real-time semantic segmentation method and system with dual-resolution interactive attention. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Semantic segmentation, a key technology in computer vision, has received widespread attention in recent years. Its goal is to enable computers to understand and interpret the category of each pixel in an image, such as people, cars, trees, etc. Its research and development are of great significance to the overall progress of artificial intelligence and machine learning. It addresses not only the technical challenges of computer science but also how to enable machines to better understand and interpret the visual information of the complex world. Research in this field continues to push the boundaries of artificial intelligence, enabling machines to process and analyze large amounts of image data more accurately and efficiently.

[0004] Research on semantic segmentation can be traced back to the early days of computer vision, when it primarily relied on traditional image processing techniques and machine learning methods. However, these early methods were limited in performance due to constraints in algorithm complexity and computational power. Driven by deep learning technology, semantic segmentation has evolved from basic image processing methods to today's highly complex and sophisticated analysis processes. This progress not only reflects the rapid development of computing power but also the continuous innovation in algorithm and model design. In particular, the application of convolutional neural networks (CNNs) in image recognition and processing has significantly improved the accuracy and efficiency of segmentation tasks.

[0005] The technological significance of semantic segmentation lies not only in its applications within computer vision but also in its contribution to the entire field of artificial intelligence. Through precise pixel-level image understanding, semantic segmentation provides machines with a capability approaching human visual perception, which is crucial for advancing machine intelligence. In practical applications, semantic segmentation has been widely used in various fields, including but not limited to autonomous driving, medical image analysis, remote sensing image processing, and human-computer interaction. In these applications, semantic segmentation not only improves efficiency but also enhances safety and reliability. For example, in autonomous driving, accurate road and obstacle recognition is essential for safe vehicle operation; in the medical field, it helps doctors diagnose and treat diseases more accurately; and in remote sensing image processing, it can be used for land cover classification and environmental monitoring.

[0006] In existing technologies, semantic segmentation techniques often suffer from the following technical problems:

[0007] (1) In semantic segmentation tasks, a trade-off usually needs to be made between processing accuracy (i.e., the accuracy of segmentation) and processing speed (i.e., the speed or real-time performance of segmentation). High-precision segmentation usually requires deep networks and complex computations, which leads to slower processing speed and is not conducive to real-time applications. Conversely, in order to improve processing speed, it may be necessary to simplify the network structure or reduce the amount of computation, but this often sacrifices segmentation accuracy.

[0008] (2) Traditional convolutional neural networks have a fixed receptive field size, making it difficult to capture both local details and global context information simultaneously. This makes it difficult for the model to balance the relationship between local features and global context when dealing with small targets or complex scenes. In addition, in order to reduce computation, semantic segmentation networks usually reduce the resolution of feature maps through downsampling operations, which leads to the loss of information about small targets and overly smooth segmentation results. Summary of the Invention

[0009] To address the aforementioned issues, this invention proposes a real-time semantic segmentation method and system based on dual-resolution interactive attention. It employs dual-resolution branches and establishes multiple bilateral connections between the two branches, using wavelet transform for lossless downsampling to achieve efficient information fusion.

[0010] In some implementations, the following technical solutions are adopted:

[0011] A real-time semantic segmentation method with dual-resolution interactive attention includes:

[0012] Obtain image information to be segmented in a given scene;

[0013] The image information is preprocessed and then input into a trained dual-resolution interactive attention real-time semantic segmentation network model to obtain the image semantic segmentation result.

[0014] The dual-resolution interactive attention real-time semantic segmentation network model first acquires shallow feature maps of the image, inputs these shallow feature maps into parallel high-resolution and low-resolution branches respectively, and performs bilateral information fusion between the two branches to acquire spatial features and contextual features of the image respectively; the acquired spatial features and contextual features are summed point-wise, and the final image semantic segmentation result is obtained through decoding.

[0015] In other embodiments, the following technical solutions are adopted:

[0016] A real-time semantic segmentation system with dual-resolution interactive attention includes:

[0017] The image acquisition module is used to acquire image information to be segmented in a given scene.

[0018] The semantic segmentation module is used to preprocess the image information and input it into a trained dual-resolution interactive attention real-time semantic segmentation network model to obtain the image semantic segmentation result.

[0019] The dual-resolution interactive attention real-time semantic segmentation network model first acquires shallow feature maps of the image, inputs these shallow feature maps into parallel high-resolution and low-resolution branches respectively, and performs bilateral information fusion between the two branches to acquire spatial features and contextual features of the image respectively; the acquired spatial features and contextual features are summed point-wise, and the final image semantic segmentation result is obtained through decoding.

[0020] In other embodiments, the following technical solutions are adopted:

[0021] A terminal device includes a processor and a memory, the processor being used to implement instructions; the memory being used to store multiple instructions adapted to be loaded and executed by the processor for the aforementioned dual-resolution interactive attention real-time semantic segmentation method.

[0022] In other embodiments, the following technical solutions are adopted:

[0023] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the above-described dual-resolution interactive attention real-time semantic segmentation method.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] (1) This invention designs a real-time semantic segmentation method with dual-resolution interactive attention by setting up a lightweight backbone network, which can effectively balance the speed and accuracy of semantic segmentation tasks, and achieve an accuracy that can be applied to practice while ensuring low computational cost.

[0026] (2) When the output of the high-resolution branch and the output of the low-resolution branch are downsampled and fused, the present invention uses Haar wavelet transform for lossless downsampling. Without losing information, the number of channels of the feature map is explicitly increased and the resolution of the feature map is reduced. Then, convolution operation is performed on the representative feature learning to filter redundant information.

[0027] (3) The context attention calibration module of the present invention can personalize the context association of each pixel, which helps to solve the problems of context mismatch and overly smooth small target segmentation results, and improves the accuracy of fine-grained image segmentation.

[0028] Other features and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0029] Figure 1 This is a flowchart of the real-time semantic segmentation method with dual-resolution interactive attention in an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the dual-resolution interactive attention neural network framework in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram illustrating the bilateral information fusion process in the implementation of this invention;

[0032] Figure 4 This is a schematic diagram of the contextual attention calibration module in an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of the recalibration module in an embodiment of the present invention. Detailed Implementation

[0034] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0035] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0036] Example 1

[0037] First, it should be noted that the dual-resolution interactive attention neural network in this embodiment refers to a real-time semantic segmentation neural network with a dual-resolution feature extraction branch as the main branch, and equipped with a bilateral fusion module and a contextual attention calibration module.

[0038] The real-time semantic segmentation method of the present invention can be applied to fields such as autonomous driving, medical image analysis, and remote sensing image interpretation.

[0039] In this embodiment, high resolution and low resolution are relative terms. The image resolution of the high-resolution branch is higher than that of the low-resolution branch, hence it is called high resolution; similarly, the image resolution of the low-resolution branch is lower than that of the high-resolution branch, hence it is called low resolution.

[0040] In one or more embodiments, a real-time semantic segmentation method with dual-resolution interactive attention is disclosed, combining... Figure 1 Taking a road scenario as an example, the specific process includes the following:

[0041] (1) Obtain the image information to be segmented in the road scene, such as the real-world image information of the road.

[0042] (2) The image information is preprocessed and input into the trained dual-resolution interactive attention real-time semantic segmentation network model to obtain the image semantic segmentation result.

[0043] In this embodiment, for training the real-time semantic segmentation network model with dual-resolution interactive attention, real-world images of the road are first collected, and the images are binarized with pixel-level annotations to determine labels, thereby forming training and testing sets, as follows:

[0044] S1.1: Real-world images of road scenes can be obtained by capturing and separating video footage. Every 10 frames of the video are extracted to obtain images, which are then input into the neural network.

[0045] S1.2: Label each pixel, determine the number of categories based on the number of basic elements in the image, assign a label to each category, fill with different colors to distinguish them, and use this label as the true value of the output image.

[0046] Based on the training set, both the input and labeled images were randomly scaled, cropped, had their boundaries filled, and flipped, thereby expanding the training set. As the training set was expanded, the accuracy improved significantly.

[0047] Each image contains a large number of pixels, and labeling each pixel is time-consuming, laborious, and prone to omissions or errors. However, a large number of images greatly helps improve accuracy. Therefore, this embodiment preprocesses the images to achieve better results with fewer images, as detailed below:

[0048] S2.1: In each training iteration, the input and labeled ground truth images are randomly scaled down or scaled up.

[0049] S2.2: If the image is larger than the original image, crop it starting from a random point; if the image is smaller than the original image, fill the boundaries and finally flip it randomly horizontally or vertically.

[0050] S2.3: The images used in each training session are different, thus expanding the training set.

[0051] Using the final constructed training and testing sets, a real-time semantic segmentation network model with dual-resolution interactive attention is trained to output accurate image semantic segmentation results. These results enable the detection of specific objects, such as:

[0052] ① Road detection: It can identify and segment road areas, helping the autonomous driving system understand the road boundaries and driving area.

[0053] ② Vehicle detection: It can identify and segment vehicles on the road, which is very important for traffic monitoring, environmental perception and decision-making of autonomous vehicles.

[0054] ③ Pedestrian detection: By separating pedestrians, the safety of autonomous driving systems can be improved and collisions with pedestrians can be avoided.

[0055] In this embodiment, the dual-resolution interactive attention real-time semantic segmentation network model first acquires shallow feature maps of the image. These shallow feature maps are then input into parallel high-resolution and low-resolution branches, respectively. Bilateral information fusion is performed between the two branches to acquire spatial and contextual features of the image. The high-resolution branch outputs spatial features, while the low-resolution branch outputs contextual features. The acquired spatial and contextual features are summed point-wise, and the final image semantic segmentation result is obtained through decoding.

[0056] Specifically, in combination Figure 2 The structure of the dual-resolution interactive attention real-time semantic segmentation network model includes two stacked basic blocks, which downsample the input image by 1 / 4 and 1 / 8 respectively. These basic blocks are used to obtain shallow feature maps of the image. These shallow feature maps are then fed into parallel high-resolution and low-resolution branches. The low-resolution branch uses ResNet as its backbone and consists of two stacked BasicBlocks and a Bottleneck convolutional block, downsampling the input image by 1 / 16, 1 / 32, and 1 / 64 respectively. The high-resolution branch is an additional branch built upon the ResNet network, using ResNet's basic blocks, including two stacked BasicBlocks and a Bottleneck convolutional block, each downsampled by 1 / 8.

[0057] The high-resolution output feature map is 1 / 8 of the original image, and has a higher resolution, so it is the high-resolution branch, which outputs spatial features; the low-resolution output feature map is 1 / 64 of the original image, and has a lower resolution, so it is the low-resolution branch, which outputs contextual features.

[0058] The output of the low-resolution branch is processed by the Context Attention Calibration (CAC) module to obtain more refined semantic context features. Finally, the context features and spatial features are summed point-wise by the summation module SUM to output the semantic segmentation result.

[0059] Based on the above structure, this embodiment first acquires shallow feature maps in the encoder stage for parameter sharing. Then, it uses a dual-resolution branch to acquire contextual and spatial features, and performs information fusion between the two branches. These features are then input into the corresponding contextual attention calibration module to obtain the final contextual and spatial features. The specific process is as follows:

[0060] S3.1: The parameter sharing stage acquires shallow features of the image and prepares for parameter sharing between the two branches in the subsequent stage.

[0061] S3.2: Input the shallow features of the image into the parallel high-resolution branch and low-resolution branch respectively. The high-resolution branch will no longer be downsampled to form high-resolution spatial features of depth, while the low-resolution features will continue to be downsampled. The feature map is gradually reduced to 1 / 64 of the original image to obtain the low-resolution contextual semantic features of the image.

[0062] S3.3: The information exchange between the high-resolution branch and the low-resolution branch is mainly composed of a dual-sided fusion module, combined with... Figure 3 , Figure 3 The left half is the low-resolution branch, which is upsampled and fused with the high-resolution branch after passing through a simple convolutional layer using bilinear interpolation; Figure 3 The right half of the image represents the high-resolution branch, which is downsampled using Haar wavelet transform and then fused with the low-resolution branch. This involves two operations: fusing the low-resolution and high-resolution branches (upsampling fusion) and fusing the high-resolution and low-resolution branches (downsampling fusion). In upsampling, the low-resolution feature map is first compressed using a 1×1 convolution, and then upsampled using bilinear interpolation to accommodate the large size of the high-resolution branch feature map. In downsampling fusion, the high-resolution feature map undergoes a special operation—lossless downsampling using Haar wavelet transform—before point-to-point summation. This process helps retain important information while reducing resolution. Therefore, the i-th high-resolution feature map H... i and low-resolution feature map L i It can be represented as:

[0063]

[0064] Among them, B H and B L T corresponds to the basic unit operations in the high-resolution branch and the low-resolution branch, respectively. L-H and T H-L These correspond to the upsampling and downsampling module operations, respectively.

[0065] Haar wavelet transform is a multi-scale analysis method. Its main principle lies in decomposing a signal or image into approximate and detailed parts of different scales, achieving multi-scale analysis through recursive downsampling and layer-by-layer processing. Specifically, for a signal of length N, the recursive formula for Haar wavelet transform is expressed as:

[0066]

[0067] Where w i These are Haar wavelet coefficients, φ i (x) are Haar wavelet basis functions. These basis functions exist in the form of averaging and difference functions, and by appropriate scaling and translation, features at different scales can be decomposed and extracted.

[0068] S3.4: The contextual attention calibration module aims to personalize the contextual association of each pixel, which helps to solve the problems of contextual mismatch and overly smooth segmentation results for small objects, improving the accuracy of fine-grained image segmentation. Combined with... Figure 4 After input, the feature map first passes through a 1×1 convolutional layer to reduce dimensionality, generating pixel features Q. Simultaneously, the module employs a pyramid structure, introducing average pooling operations at different scales to eliminate redundant information. This cascaded structure yields a multi-scale feature, which is then passed through two 1×1 convolutional layers to generate two forms of contextual representations K and V. Matrix multiplication is then performed between Q and K, and a softmax layer is used to generate the correlation between pixels and their context, specifically:

[0069]

[0070] Where Θ i,j Q represents the feature of the i-th pixel. i With the j-th context feature K j The correlation degree is then determined. Furthermore, matrix multiplication is performed on V and Θ to obtain the processed semantic context features, which are then passed through the recalibration module to generate refined semantic context features.

[0071] The recalibration module uses the tanh function, with the following structure: Figure 5 As shown, a residual structure is employed. The input undergoes two convolutional operations: a Conv-BN-ReLU layer and a separate convolutional layer. Then, the tanh function is used to extract valid information, which is finally multiplied and added back to the input to obtain the final output. This approach aims to eliminate redundant information and emphasize valid information within the context, especially information related to boundaries or small targets. Its specific structure can be defined as follows:

[0072]

[0073] Where I and O represent input and output, ConvX1 and ConvX2 represent two convolution operations, and α is the recalibration factor.

[0074] S3.5: Finally, the refined semantic context features and spatial feature inputs from the recalibration module are summed element-wise to obtain the final output.

[0075] In this embodiment, the context features and spatial features output by the high-resolution branch and the low-resolution branch are summed pointwise, upsampled, and then input into a preset decoder to output the final image semantic segmentation result. Inference and matching are then performed to complete the entire semantic segmentation process.

[0076] This embodiment utilizes a dual-resolution interactive attention neural network to solve the semantic segmentation problem. Leveraging the advantages of two-stream network feature extraction and drawing inspiration from the concept of sharing shallow features in Fast-SCNN, multiple bilateral connections are established between the two branches. Lossless downsampling is performed using wavelet transform to achieve efficient information fusion. Furthermore, a separate self-attention module is designed for the role of each branch, enabling efficient mining of contextual and spatial information. Experiments demonstrate that the proposed network strikes a good balance between segmentation speed and accuracy.

[0077] Example 2

[0078] In one or more embodiments, a real-time semantic segmentation system with dual-resolution interactive attention is disclosed, specifically including:

[0079] The image acquisition module is used to acquire image information to be segmented in a given scene.

[0080] The semantic segmentation module is used to preprocess the image information and input it into a trained dual-resolution interactive attention real-time semantic segmentation network model to obtain the image semantic segmentation result.

[0081] The dual-resolution interactive attention real-time semantic segmentation network model first acquires shallow feature maps of the image, inputs these shallow feature maps into parallel high-resolution and low-resolution branches respectively, and performs bilateral information fusion between the two branches to acquire spatial features and contextual features of the image respectively; the acquired spatial features and contextual features are summed point-wise, and the final image semantic segmentation result is obtained through decoding.

[0082] The specific implementation methods of the above modules have been described in Example 1, and will not be detailed here.

[0083] Example 3

[0084] In one or more embodiments, a terminal device is disclosed, including a server. The server includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the real-time semantic segmentation method with dual-resolution interactive attention as described in Embodiment 1. For the sake of brevity, further details are omitted here.

[0085] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0086] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0087] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0088] Example 4

[0089] In one or more embodiments, a computer-readable storage medium is disclosed, wherein a plurality of instructions are stored, the instructions being adapted to be loaded by a processor of a terminal device and executed by the real-time semantic segmentation method of dual-resolution interactive attention described in Embodiment 1.

[0090] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A real-time semantic segmentation method with dual-resolution interactive attention, characterized in that, include: Obtain image information to be segmented in a given scene; The image information is preprocessed and then input into a trained dual-resolution interactive attention real-time semantic segmentation network model to obtain the image semantic segmentation result. The dual-resolution interactive attention real-time semantic segmentation network model first obtains shallow feature maps of the image, inputs these shallow feature maps into parallel high-resolution and low-resolution branches respectively, and performs bilateral information fusion between the two branches to obtain spatial features and contextual features of the image respectively; the obtained spatial features and contextual features are summed point-wise, and the final image semantic segmentation result is obtained through decoding; The output of the low-resolution branch is input to the context attention calibration module, and then summed with the output of the low-resolution branch in a point-wise direction; the context attention calibration module is specifically: The input feature map is first passed through a convolutional layer to reduce its dimensionality, generating pixel features Q; Simultaneously, the feature map obtains a multi-scale feature through cascaded average pooling units, and then passes it through two convolutional layers to generate two forms of contextual representations K and V respectively; the features Q and K are multiplied by a matrix, and a softmax layer is used to generate the correlation between pixels and context. Matrix multiplication is performed on V and Θ to obtain the processed semantic context features, which are then passed through the recalibration module to generate refined semantic context features. Bilateral information fusion is performed between the two branches, specifically as follows: When upsampling and fusing the outputs of the low-resolution branch and the high-resolution branch, channel compression is performed on the output of the low-resolution branch, and bilinear interpolation is used for upsampling to adapt to the feature map size of the high-resolution branch output. When downsampling and fusing the outputs of the high-resolution branch and the low-resolution branch, the output feature map of the high-resolution branch is losslessly downsampled using wavelet transform, which reduces the resolution while retaining important information.

2. The real-time semantic segmentation method with dual-resolution interactive attention as described in claim 1, characterized in that, The low-resolution branch uses ResNet as its backbone network, while the high-resolution branch is an additional branch built on top of the ResNet network, using basic blocks from ResNet.

3. The real-time semantic segmentation method with dual-resolution interactive attention as described in claim 1, characterized in that, The recalibration module specifically comprises: Where I and O represent input and output respectively, ConvX1 and ConvX2 represent two convolution operations, and α is the recalibration factor.

4. The real-time semantic segmentation method with dual-resolution interactive attention as described in claim 1, characterized in that, The final output is obtained by element-wise summing the refined semantic context features obtained through the recalibration module with the input feature map.

5. A real-time semantic segmentation system with dual-resolution interactive attention, characterized in that, include: The image acquisition module is used to acquire image information to be segmented in a given scene. The semantic segmentation module is used to preprocess the image information and input it into a trained dual-resolution interactive attention real-time semantic segmentation network model to obtain the image semantic segmentation result. The dual-resolution interactive attention real-time semantic segmentation network model first obtains shallow feature maps of the image, inputs these shallow feature maps into parallel high-resolution and low-resolution branches respectively, and performs bilateral information fusion between the two branches to obtain spatial features and contextual features of the image respectively; the obtained spatial features and contextual features are summed point-wise, and the final image semantic segmentation result is obtained through decoding; The output of the low-resolution branch is input to the context attention calibration module, and then summed with the output of the low-resolution branch in a point-wise direction; the context attention calibration module is specifically: The input feature map is first passed through a convolutional layer to reduce its dimensionality, generating pixel features Q; Simultaneously, the feature map obtains a multi-scale feature through cascaded average pooling units, and then passes it through two convolutional layers to generate two forms of contextual representations K and V respectively; the features Q and K are multiplied by a matrix, and a softmax layer is used to generate the correlation between pixels and context. Matrix multiplication is performed on V and Θ to obtain the processed semantic context features, which are then passed through the recalibration module to generate refined semantic context features. Bilateral information fusion is performed between the two branches, specifically as follows: When upsampling and fusing the outputs of the low-resolution branch and the high-resolution branch, channel compression is performed on the output of the low-resolution branch, and bilinear interpolation is used for upsampling to adapt to the feature map size of the high-resolution branch output. When downsampling and fusing the outputs of the high-resolution branch and the low-resolution branch, the output feature map of the high-resolution branch is losslessly downsampled using wavelet transform, which reduces the resolution while retaining important information.

6. A terminal device comprising a processor and a memory, the processor for implementing instructions; the memory for storing multiple instructions, characterized in that, The instructions are adapted to be loaded by a processor and executed by the real-time semantic segmentation method of dual-resolution interactive attention as described in any one of claims 1-4.

7. A computer-readable storage medium storing a plurality of instructions, characterized in that, The instructions are adapted to be loaded by the processor of the terminal device and executed by the real-time semantic segmentation method of dual-resolution interactive attention as described in any one of claims 1-4.

Images